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I. INTRODUCTION 


A. BACKGROUND 

1. Security Assistance 

Security assistance, as defined in its simplest terms. concerns the transfer of 
military and economic assistance products and services to foreign governments through 
sale, grant, lease or loan. Promulgated into law by the Foreign Assistance Act of 1961. 
security assistance includes the Foreign Military Sales (FMS) program. the Economic 
Support Fund (ESF), International Military Education and Training (IMET). 
Peacekeeping operations, and the Commercial Export Sales program. Currently the 
Secretary of State has statutory responsibility for supervision and general direction of 
security assistance programs and the Secretary of Defense is responsible for 
implementing military programs through the Defense Security Cooperation Agency 
(DSCA).! 

Ze Post Cold War Security Environment 

Recent world events such as the break-up of the former Soviet Union and Gulf 
War have caused a modification of U.S. security assistance goals. According to a 1997 


U.S. General Accounting Office (GAO) report, 


! Until October 1, 1998, DSCA was known as the Defense Security Assistance Agency (DSAA). 
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We now face challenges involved in NATO expansion, assisting the 
transition of former Communist countries, integrating China into the world 
economic system, reforming the United Nations and the World Bank, 
building a credible World Trade Organization, and attacking the menace 

of illegal drugs and AIDS. These events necessitate a careful rethinking of 

the programs, approaches, and activities at the U.S. government’s disposal 

to advance its interests. At the same time, the current environment also 

affords the chance to reassess programs and activities on their merits and 

determine if they are relevant in today’s world (GAO, 1997, p. 1). 

Security assistance programs such as IMET focus upon the enhancement of 
democratic institutions worldwide. With the new emphasis on providing support for 
emerging democracies, a challenge facing policy makers in the Post Cold War 
environment is to assess the effectiveness of current security assistance programs. 

> IMET and E-IMET 

a. The IMET Program 

Section #541 of the Foreign Assistance Act (FAA) of 1961 defines IMET 
as a key component of U.S. security assistance that provides professional military 
education (PME) on a grant basis to students from allied and friendly nations (GAO, 
1992). The Congressional Presentation for Foreign Operations (CPFO) states that the 
overall goal of the program is to “improve the ability of participating foreign countries to 
utilize their resources” thereby contributing to greater self-reliance (CPFO, 1998, p. 19). 
The intention is to provide a wide range of benefits to the U.S in terms of collective 
security, stability, and peace. 

IMET is a program that, for a relatively modest investment, presents 
democratic alternatives to key foreign military and civilian leaders (CPFO, 1998). The 


IMET program’s FY98 budget request was for a mere 50 million dollars, less than the 
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cost of one U.S. Air Force F-22 fighter jet.2 In fact. funding for the entire international 
affairs budget (Federal Budget Function #150), which totaled $18.1 billion in fiscal vear 
1997, constitutes only one percent of the federal budget and just three to four percent of 
federal discretionary funding (GAO, 1997). Funding for the international affairs account. 
however, has declined in real terms about 6% annually from 1992 to 1997 (GAO. 1997). 
Nineteen ninety-seven funding was 23% lower than 1992 levels (GAO. 1997). 
Additionally, Function 150 1s slated to receive cuts of over 13% 1n real terms by 2002 
(GAO, 1997). 

Although international affairs programs such as IMET comprise a fraction 
of federal discretionary funding, “they are designed to influence world political and 
economic agendas” (GAO, 1997, p. 2).° 

b. The Expanded IMET Initiative 

IMET is expanding and taking new directions 1n response to the changing 
global political scene. Significant changes have taken place to align the program 
objectives with U.S. foreign policy interests in the post-Cold War environment. In 199] 
the Congress amended the 1961] Foreign Assistance Act with the Foreign Operations 
Appropriations Act, earmarking $1 million of IMET funds to expand the IMET program. 


This initiative expanded existing IMET courses and implemented new training programs 


2 Currently the Air Force estimates that the F-22 fighter jet will cost roughly $87 million each (Cole. 
1993): 


3 According to the Congressional Budget Office (CBO), 1997 discretionary outlays totaled approximately 
$550 billion (CBO, 1997). Therefore, funding for Budget Function #150 equaled roughly 3% of al] 1997 
discretionary expenditures (GAO, 1997). The 1997 IMET appropriation of $43.5 million only absorbed 
.24% of Function #150 funding, or less than .01% of total 1997 discretionary spending. 
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focusing on civilian control over the military, respect for human rights, and responsible 
defense resource management (U.S. Congress, 1991). Most importantly, however, the 
traditional IMET student population was “enlarged to include civilian personnel from 
non-defense ministries, officials from the legislative branch who deal with military 
matters, and personnel from nongovernmental agencies having defense-related interests” 
(Cope, 1995, p. 6). By 1998 approximately 25-30% of total IMET funding was dedicated 
to the E-[MET initiative (Verry, 1998). 

G IMET and E-IMET: A Low-Cost Alternative 

In 1990, DSCA claimed that IMET is an effective, low-cost component of 
the global U.S. security assistance effort (Larson, 1993). “The IMET program has 
become one of the most cost-effective means for promoting U.S. interests and advancing 
security in the Asian-Pacific-Indian Ocean region and throughout the world,” confirmed 
the Commander-in-Chief Pacific in 1993 (Larson, 1993, p. 14). There is a belief among 
existing security assistance providers that the relatively small cost of programs such as 
IMET and E-IMET provides a return for U.S. objectives on a global scale and that over 
the years this return is far greater than the original investment. 

4. Results Oriented Emphasis for Security Assistance and Government 


According to SECSTATE Christopher in 1993, 


The watchwords for security assistance are now empowerment, partner- 
ship, and effectiveness. Our foreign assistance programs will be result- 
oriented, not expenditure-oriented. Where scarce development resources 
cannot be used effectively, our assistance programs should be reduced or 
redirected. Our development assistance should be judged not on the basis 
of funds obligated but on the basis of results achieved, and the same 
applies to security assistance (Christopher, 1993). 


Richard Moose, Under Secretary for Management 1n 1994. echoed the Secretary's 
words and cited the fact that the Foreign Assistance Act (l: AA) of 196] has become laden 
with competing and conflicting goals and objectives. 

In a time of reduced budgets, it is more important than ever that our 

programs be reformed and our scarce resources be targeted accurately and 

used effectively. We cannot afford waste and inefficiency. We [must] 

shift our focus away from how we do things and toward what we are 

trying to accomplish—away from process toward results. We will set 


clearly understood goals by which our programs can be measured. and we 
will demand measurable results (Moose, 1994). 


Two particular initiatives that are national in scope and demonstrate overall 
government emphasis on performance measurement are the Government Performance 
and Results Act (GPRA) of 1993 and the Service Efforts and Accomplishments (SEA) 
Reporting initiative of the Governmental Accounting Standards Board (GASB). 

Signed by President Clinton in 1993, the GPRA institutionalized the federal 
government’s commitment to developing and deploying effective performance measure- 
ment and performance management systems based on agency goals and objectives. The 
Results Act requires all government agencies (except the Central Intelligence Agency and 
some very small agencies) to establish top-level goals and objectives. define how they 
intend to achieve those goals, and demonstrate how they will measure agency and 
program performance in achieving those goals (GPRA, 1993). Agencies are required to 
submit annual performance reports to Congress starting no later than March 31. 2000. 


comparing actual performance against agency plans (Epstein, 1996). 


While GASB actions do not have the force of law like GPRA, their SEA 
Reporting initiative is recognized by the American Institute of Certified Public 
Accountants for use in public financial reporting (Epstein, 1996). With SEA reporting, 
GASB has moved beyond standard monetary accounting into the realm of performance 
measurement. The SEA framework stresses outcomes and efficiency as well as inputs 
and outputs (Epstein, 1996). In short, GASB has recognized “that public accountability 
cannot be achieved only through traditional reporting of monetary information” (Epstein, 


1996, p. 62). Performance reporting is also needed. 


B. STATEMENT OF PROBLEM 

In this time of reduced budgets and evolving security assistance and performance 
review initiatives, the DSCA and the Departments of State (DoS) and Defense (DoD) are 
faced with the challenge of measuring the effectiveness and performance of their security 
assistance programs. How this is to be done remains an open question. 

A 1992 GAO report exploring post-Cold War security assistance program changes 
highlighted the fact that the DoD and DoS had not yet agreed on a format or the type of 
information to be included in an assessment of the overall IMET program (GAO, 1992). 
DSCA initially noted that the success of the E-IMET initiative could not be measured for 
“at least another 5 years” (GAO, 1992). GAO recommended that the Departments of 
State and Defense jointly develop a system to evaluate the effectiveness of the E-IMET 


program. 


c RESEACH QUESTIONS 


The primary research question of this thesis 1s: What could a performance 


evaluation system developed for the E-[MET program look like? 


Subsidiary research questions supporting this issue are: 


to 


oe) 


What are the challenges to performance evaluation of public programs? 


What factors unique to the political environment necessarily limit the 
application of rational-analytic assessment models when measuring costs 
and benefits of current security assistance programs? 


Can a system be developed that balances the reality of the political 
bureaucratic process with the rationality of economic cost-benefit 
analysis? 


What are the implications of implementing a performance evaluation 
system on current Security assistance programs? 


D. OBJECTIVE 


The objective of this thesis is to address how the DSCA and the Department of 


Defense can tackle the challenge of measuring the effectiveness and performance of their 


security assistance programs. Specifically. it will investigate potential methods for 


performance evaluation of the E-IMET program. 


E. METHODOLOGY 


The following approach will be used during this study: 


oy) 


Identify a continuum of policy and program evaluation paradigms within 
the public sector performance evaluation literature. 


Identify the key stakeholders of the E-IMET security assistance program 
and the objectives and goals of each. 
Develop a Global Objectives Hierarchy tor U.S. security assistance. 


| 
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4. Interview members of the IMET and E-IMET management staff to 
validate the proposed objectives hierarchy. 


Dy Develop performance indicators for the E-IMET program. 


6. Address the impact of the proposed evaluation methodology on the current 
E-IMET program. 


ae Recommend a course of action that brings the E-IMET program in line 
with the federal government’s mandate for performance review. 


F. ORGANIZATION OF STUDY 

Chapter I, /ntroduction, will present the focus of this study. It will introduce the 
IMET and E-IMET security assistance programs and discuss the challenges presented by 
the changing international security environment and the 1993 Government Performance 
and Results Act. 

Chapter II. Literature Review, will present the purpose, current utilization, and 
future use of performance measurement in government. It will also explore the 
challenges of public sector performance evaluation and conclude with a description of the 
process that will be used to measure the effectiveness of the E-IMET program. 

Chapter HI, Methodology, will outline the research methodology used to conduct 
this study. 

Chapter IV, Data Collection, will identify the stakeholders of the E-IMET 
program in order to inform the objectives setting process and enable the development of 


performance indicators. 


Chapter V. Analysis. Part I. will identify each stakeholder’s security assistance 
objectives in order to form a Global] Objectives Hierarchy and create an objective baseline 
for future performance measurement efforts. 

Chapter VI. Analysis: Part [7]. will propose various performance indicators for the 
E-IMET program. 

Chapter WII. Conclusion. will summanze the researchers findings and 
recommend a course of action for the implementation of an E-IMET pertormance 


evaluation system. 
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IT. LITERATURE REVIEW 


x: PERFORMANCE EVALUATION IN THE PUBLIC SECTOR 

Much of the existing literature pertaining to performance evaluation in the public 
sector explores the purpose, current utilization, implementation problems. and future use 
of performance measurement 1n government. In order to fully understand performance 
evaluation one must clearly define a measurement system. differentiate between process 
and results-oriented systems, recognize the purpose and need for a system, and appreciate 
the forces affecting its application. 


ile Measurement System Defined 


According to Halachmi and Bouckaert. a measurement system “consists of 
practices, procedures, criteria, and standards that govern the collection of data (input). the 
analysis of the data (throughput), and the compilation of the results into quantitative or 
qualitative forms (output)” (Halachmi and Bouckaert, 1996, p. 2). Greiner cites a 1992 
GAO report that further breaks performance measurement into subsystems: 

Program performance measurement 1s commonly defined as the regular 


collection and reporting of a range of data including: 


Inputs (dollars, staff, and materials). 


. Workload or activity levels, 

* Outputs or final products. 

. Outcomes of products or services, and 

° Efficiency-cost per unit output or output per unit cost. sometimes referred 


to as “productivity (Greiner, 1996, p. 2). 


5 


This systems definition, including inputs, throughput, outputs, and outcomes, 
must also recognize factors that influence each subsystem. For instance, the input side is 
influenced by criteria and standards, the methods and frequency of collecting the data, 
and who is in charge of the collection (Greiner, 1996). The analysis, or throughput, is 
influenced by the techniques used to transform raw data into measures of performance. 
The nature of the results and how those results are communicated to users within an 
organization influence output (Greiner, 1996). Greiner concludes that the measurement 
of government performance requires the collection and reporting of data at each point 
along the causal chain—inputs, throughput (processes), outputs, and outcomes (Greiner, 
1995). 

Depending on the organization’s objectives, agencies may have to use more than 
one kind of measurement of performance. Indeed, the multidimensional nature of many 
public sector activities demands the use of multiple performance measures. Thus, clearly 
identified agency objectives are central to the development of any performance 
measurement system. 

2 Differentiating Between Process and Results-Oriented Systems 

The literature distinguishes between measurement systems that focus on results 
from those that focus on process. If desired results is the target, then process 
specifications are derived from the desired results (Halachmi and Bouckaert, 1996). With 
this focus, the purpose of the measurement system is to assess the attainment of the 
desired results, or oufcomes, based on clearly defined agency objectives. Outcome 
measurement can be used in a retrospective role--to determine whether the expected 
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benefits of a public sector program have materialized (Smith, 1996). It can also be used 
In a prospective role--to guide future resource allocation decisions. A results-oriented 
system, therefore, attempts to measure the outcomes--or effectiveness--of a given 
program, and 1s the focus of this thesis. 

A process-focused performance measurement system assesses efforts to 
reengineer agency processes in order to achieve greater efficiency. The focus is on the 
causal chain from inputs to outputs. Process-focused systems measure the productivity of 
inputs used to create owfputs in various program processes. It is a “clinical approach” to 
management; one that 1s more concerned about the procedural aspects of the operation 
than the results of the operation (Halachmi and Bouckaert. 1996, p. 4). 

>: Purpose of Performance Measurement 

Hatry states that the “ultimate purpose of performance measurements is to uSe the 
measurement information to help make improvements—whether to expand. delete, or 
modify programs” (Hatry, 1996, p. villi). Greiner lists several practical reasons for 
implementing performance measurement: improvement of performance, of motivation. of 
communications, and of accountability (Greiner, 1996). 

Improving performance is the most important application of performance 
measurement. “Performance measures can be used to enhance planning. budgeting, and 
service delivery by providing timely feedback of results” (Greiner, 1996, p. 19). The 
availability of performance information makes continuous reassessment and tmprovement 
possible. Enhancing motivation is possible when performance information 1s linked to 


Incentives to stimulate emplovees. Performance information also encourages 


communication between managers, government officials, and elected representatives 
(Greiner, 1996). Finally, performance data can be used to show various interest groups 
just what they are getting for their tax dollars. Indeed, regular performance reports can be 
viewed “as a way to keep the public involved in the process of government” (Greiner, 
1996, p. 20). 

An increasing number of critics have recognized that the federal government has 
developed two defects that are central to its existence: (a) it does not know how to tell 
whether many of the things it does are worth doing at all; and (b) when it does decide that 
something is worth doing, it does not know how to create and sustain a program capable 
of achieving the results it seeks (Quade, 1984). In other words, what government lacks 1s 
some sort of regulator that tells it when a program has ceased to be effective or could be 
made more efficient. With increasing emphasis on providing more value for money with 
limited resources, an assessment of performance is needed to inform the decision-making 
process and may ultimately become a matter of government survival (Greiner, 1996). 

Many authors agree that information and knowledge, not land, labor, and capital, 
are the key resources in a “knowledge society.” Managing change and ensuring that 
informed decisions are made quickly requires timely and complete information on 
performance. “Without adequate measurement, managers and employees within the 
agency, other agencies and oversight bodies, interest groups, the public, or the courts may 


not be able to document changes in performance” (Halachmi and Bouckaert, 1996, p. 3). 


4. Who Needs the Information Provided by Performance Measurement 

The public management literature indicates that performance data is useful to 
many parties. Program managers need regular project performance information to 
identify where their programs are doing well, where not well, and to help identify 
improvement actions (Halachmi and Bouckaert, 1996). Staff needs this information to 
determine how well they are doing. The media and the public also need performance 
information in order to fully participate in a democratic government. 

In government, performance measurement can be used by the legislature and 


within the staff function of various government agencies. Halachmi and Bouckaert write: 


It can be a budget decision tool, determining where to allocate scarce 
funds to achieve the greatest impact. It can be used by senior managers to 
fine-tune the agency through corrective measures or to provide front-line 
employees with feedback about their performance. It can provide 
information for planning by suggesting which operations need to be re- 
engineered to be more effective and efficient. It can be a communications 
tool, informing clients of the level of service achieved. For clients, it can 
be a feedback tool, allowing them an opportunity to indicate how services 
meet or do not meet their needs (Halachmi and Bouckaert, 1996. p. 6). 


The literature also emphasizes that all parties must understand the limitations as to 
what these data tell and what they do not tell, as performance measurement systems have 
many potential problems. 

ay Current Public Sector Utilization 

A 1992 GAO survey of 102 federal agencies found that all of them collected a 


wide variety of performance measures (Greiner. 1996). However, according to Greiner. 


Such statistics often give a false impression of the scope and depth of the 
performance measurement efforts underway. The reported measurement 
activities often focus primarily on input and workload measures, and in 
many instances they involve only limited number of agencies and 
applications (Greiner, 1996, p. 14). 


The 1992 GAO report agreed, stating that, although most agencies employed a variety of 
performance measures, substantive use of such information was limited. The GAO 
survey concluded that many federal agencies only collected outcome data for 
informational purposes and few used performance measures to facilitate the achievement 
of long-term objectives (Greiner, 1996). 

Other authors have arrived at similar conclusions. Wholey and Hatry observe that 
“most reports on government performance still focus on resource expenditures and the 
quantity of services delivered.:.; few government agencies provide timely information on 
the quality and outcomes of their major programs” (Wholey and Hatry, 1992, p. 604). 
Cave, Kogan, and Smith add that in the public sector there is often “a need for greater 
attention to quality of output and customer service, and levels of service quality cannot be 
adequately proxied by physical output measures” (Cave, Kogan, and Smith, 1990, p. 10). 

The performance evaluation literature concludes that current federal agencies have 
underutilized existing measurement systems. While performance data exists, agencies 
fail to use the information to motivate long-term mission accomplishment (Wholey and 


Hatry, 1992). 


6. Problems Associated with Public Sector Performance Evaluation 

Understanding the lack of public sector performance evaluation requires an 
examination of the institutional, technical, and financial obstacles associated with any 
measurement system. These “types of obstacles have hampered (and continue to hamper) 
wider acceptance and application of performance measurement in the public sector” 
(Greiner, 1996, p.16). 

a. Institutional Obstacles 

Several common characteristics of American government contribute to the 
limited use of performance measurement in the public sector. First, political pressures 
often outweigh the impact of data in the bureaucratic decision-making process (Greiner. 
1996). All too often performance measurement efforts carry less influence than the 
actions of powerful elected officials, interest groups, and agency officials. 

The presence of multiple self-interested constituencies also makes it 
difficult—if not impossible—‘“to reach agreement on service goals and objectives that 
must proceed performance measurement” (Greiner, 1996, p. 21). The impact of multiple 
constituencies on public policy receives extensive treatment in the performance 


evaluation literature. For example, Hitch recognized that: 


National objectives can only be some combination or distillation of the 
objectives of people who comprise (or rule) the nation: and we should 
learn to be skeptical and critical of the verbalizations and rationalizations 
that pass for national objectives (Hitch, 1960). 


A 1993 Congressional Budget Office (CBO) report concluded that, “if a 


program’s objectives cannot be determined, performance measures will always be 
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ambiguous, if not superfluous” (CBO, 1993, p. 5). Unlike the corporate model with a 
bottom line, differing objectives of multiple constituencies cannot all be measured via a 
single performance index (Boschken, 1994). In a plural society diverse constituencies 
want different things from their public services. They demand performance information 
concerning multiple, and often conflicting, objectives. Implementing a performance 
measurement system that satisfies each constituency is often impossible and represents a 
major obstacle to public sector performance evaluation. 

Utilization of performance measures has also been unfamiliar to the 
management approach of many public sector managers. These managers are often 
unaccustomed to using quantitative data to aid decision-making. A related institutional 
obstacle to the acceptance of performance measurement Is a government's “political self- 
confidence” (Greiner, 1996, p. 17). Are administrators and managers willing to accept 
both bad and good reports? “The prospect of accurate performance information and 
increased accountability can be very threatening to some elected officials, managers, line 
personnel, and interest groups” (Greiner, 1996, p. 17). 

These factors combine to form a lack of institutional readiness for 
performance evaluation. 

b. Technical Obstacles 

Numerous technical obstacles inhibit performance evaluation. Like 
institutional obstacles, these technical challenges represent significant impediments to the 


measurement process. 
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At the most basic level is the lack of a common definition of 
productivity—a key element of performance measurement. Greiner points out that no 
consensus exists on the definition of government productivity, let alone how it is to be 
measured (Greiner, 1996). Both measures of efficiency and effectiveness have 
inappropriately been identified as definitions of productivity. 

Differentiating between output and outcome measures when measuring 
organizational productivity is a common problem. Output measurement traditionally 
refers to the quantity of the good or service provided, and is closely related to the concept 
of efficiency (Smith, 1996). Many agree that there is no shortage of quantifiable output 
indicators. Largely dependent on output, outcome refers to “the impact on society of a 
particular public sector activity,” and is closely related to the concept of effectiveness 
(Smith, 1996, p. 1). Unlike output, outcome is far more difficult to measure. As Smith 
points out, “although most public sector auditing bodies pay lip-service to the desirability 
of addressing effectiveness issues, they have found it difficult to put into practice the 
principles of outcome measurement” (Smith, 1996, p. 2). One reason that helps explain 
the difficulties with outcome measurement is the fact that many costs and benefits of 
various programs are non-quantifiable. The difficulties of systematically measuring the 
outcomes of governmental programs, like security assistance, greatly limit the use of 
performance measurement techniques. 

Another technical obstacle to the adoption of performance measurement 
systems is the fact that government services typically require several different measures 
to adequately characterize their performance (Smith. 1996). The multifaceted nature of 
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individual government programs often means that performance cannot be captured by 
only one or two measures. Information overload, caused by multidimensional programs, 
often overwhelms potential users (Smith, 1996). 

Other important obstacles limit the adoption and use of performance 
measurement systems. The lack of standards for judging government performance limits 
the utility of performance data. Agencies need a ready benchmark to measure their 
performance data against. Without it they may not be motivated to corrective action. 
Making performance data available in a timely fashion is also difficult. If data is to be 
useful, performance feedback must occur often and be consistent. In reality, however, the 
data collection process often involves tedious delays (Greiner, 1996). 

c Financial Obstacles 

Implementation of performance measurement systems can be very costly. 
In an era of budgetary constraint the availability of resources (staff trme and dollars) for 
performance evaluation is often limited. As Wholey and Hatry point out, “the cost of 
performance monitoring must always be balanced against the value of performance 
monitoring 1n improving government performance and credibility” (Wholey and Hatry, 
1992, p. 609). 

de Summary 

Performance measurement in public policy is complex and problematic. Public 
sector operations are multidimensional, possess multiple or poorly defined objectives, 
lack comparative standards, and often result in non-quantifiable outcomes. Thus, public 
sector performance evaluation cannot be solely scientific. It must take into account the 
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political pressures of powerful stakeholders, the lack of institutional readiness. and 
various technical obstacles. Attempts at measurement must concentrate not only on 
inputs, but also systematically on the entire value chain, including throughput and 
Outputs. Public agencies must ultimately attempt to measure the effectiveness. or 
outcomes, of their programs. 

Although some areas continue to pose problems for performance evaluation in 
government, many government functions are yielding to performance measurement 
systems. A spectrum of approaches to performance evaluation of public programs 
emerges. Anchoring one end of the spectrum 1s the “technically rational paradigm” that 
treats performance measurement as an objective, scientific aid to decision-making. 
Anchoring the opposite end of the spectrum is the “politically rational paradigm” that 
emphasizes the effect of political power on the measurement of performance. Each 
paradigm employs various analytical techniques to quantify data and measure 


performance. 


B. PERFORMANCE MEASUREMENT PARADIGMS 

A paradigm is “a systematic statement of the basic assumptions. concepts. and 
propositions employed by a school of analysis” (Allison, 1971, p. 32). Approaches to 
public sector performance evaluation form a continuum from the “technically rational 


paradigm” to the “politically rational paradigm.” 


1. The “Technically Rational Paradigm” 

The most widely used approach to public sector performance évaluation is the 
technically rational paradigm. Central to the application of this paradigm is the 
assumption that program objectives are clearly stated and costs and resulting benefits are 
measurable. Being able to weigh, measure, and quantify is essential to this type of policy 
analysis. Objectives consensus is required before any measures of effectiveness are 
developed. Its dominant appeal is that it would discipline public choice so that scarce 
public resources are rationally allocated to their highest valued uses (Byme, 1987). 
Examples of well-documented technically rational techniques include economic cost- 
benefit analysis, performance ratios, and the linear programming model. This paradigm 
also includes traditional operations research techniques that address the problem of 
multidimensionality such as data envelopment analysis and multiattribute utility theory 
(Greiner, 1996). 

Users of technically rational techniques, like cost-benefit analysis, within 
government fall generally into two groups; those who recognize decisions as necessarily 
imperfect and treat techniques like cost-benefit analysis as an organizing rather than a 
calculative framework for addressing problems, and those which see the technique as a 
way to establish formal criteria in government decision making (Byrne, 1987). Thus, a 
split emerges between those who use technically rational techniques as an informative aid 
to decision-making, and those who advocate their rigorous application in government. 
Critical review, however, suggests that the ngorous application of techniques like cost- 
benefit analysis has generated numerous problems. 
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One obvious problem is quantifying intangibles like quality of life or the value of 
a human life. As stated earlier, to proceed logically in this paradigm one must be able to 
weigh, measure, and quantify. There is an inherent bias to favor elements that are 
quantifiable—even if they are not relevant to the critical questions being asked. This 
shortcoming surfaces in the area of public policy. Because many public programs 
produce benefits that are not quantifiable, only their dollar costs are readily calculated. 
The result is an “impression of accuracy that does not exist but also the notion that solely 
that which is quantifiable 1s significant” (Hoos, 1972, p. 139). Applying economic cost- 
benefit analysis to public sector decisions risks the “distortion, and ultimately. superficial 
treatment of matters of the deepest consequence” (Hoos, 1972. p. xxix). The inevitable 
consequence of this urge to scientific objectivity is the tendency to treat the matter under 
study as though it was primarily economic and thus can be judged through technically 
rational analysis. 

The existence of numerous political constituencies raises the issue of multiple 
objectives. Application of the technically rational paradigm assumes a consensus on 
objectives. In order to technically measure performance one must have a clear idea what 
it is that they are measuring, and to what their results will be compared. Getting 
competing constituencies to agree on a single set of objectives is often difficult. But 
agreement on a single set of objectives is only half of the problem. What. then. should 
the criteria be for selecting one alternative over another and for deciding whether 
performance has met the stated objectives? Assuming that quantifiable data exists. 


simply calculating a performance ratio does not yield the ultimate answer. The reality of 


public sector policy analysis is that political pressures play a key role in the setting of 
objectives and the selection of performance criteria. 

The true value, and main difficulty, of technically rational methods such as cost- 
benefit analysis is its forced concentration on the measurable elements rather than the 
whole. Through this “piece-meal fragmentation” or “‘sub-optimization,” analysts juggle 
the parts that they can handle (Hoos, 1972, p. 138). While measures of overall 
effectiveness remain difficult to quantify, and thus outside the realm of rigorous cost- 
benefit analysis, pieces of the overall puzzle may be within grasp. The goal, then. ought 
not to be a definitive solution but a reasonable one, founded on an informed under- 
standing of the measurable sub-elements of the larger objective. 

Zz The “Politically Rational Paradigm” 

The Founding Fathers’ concerns for liberty resulted in a tripartite division of 
powers within the national government. This deliberately pluralist formulation is the 
source of many of the difficulties surrounding performance measurement in the public 
sector. The assessment of effectiveness requires specification of objectives against which 
outcomes can be measured. But whose objectives should be considered? Technically 
rational performance measurement models have “a tradition of defining ‘performance’ 
from the perspective of one stakeholder in particular” (Bovaird, 1996, p. 146). As a 
result, technically rational measurement systems are often totally at odds with their 


political environment. 
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a. Multiple Objectives 

Public agencies that are owned and funded by democratic governments 
operate under political authority (Rainey, 1991). Multiple constituencies (stakeholders). 
such as interest groups, elected and agency officials, the media, legislative bodies. and the 
general public, heavily influence the exercise of that authority. The presence of multiple 
constituencies within the political system results in the pursuit of multiple. intangible. and 
often, conflicting, objectives (Rainey, 1991). As a result, performance evaluation 
becomes highly subjective. 

To reach consensus on objectives, against which outcomes should be 
measured, all groups affected by the policy must share a common interest that outweighs 
their particular interests. The relative political power of each group, however. determines 
their impact on the objectives-setting process. Objectives consensus, therefore. is not 
objective or equally weighted, but rather a result of the power and bargaining skill among 
various stakeholders (Allison, 1971). A single set of objectives “results from 
compromise, conflict, and confusion [among] officials with diverse interests and unequal 


influence” (Allison, 1971, p. 162). As Smith points out, 


It is rare to find consensus about what constitutes the objectives of a public 
sector program. Thus, in making a choice of outcome measures, the 
outcome measurer 1s implicitly reflecting the objectives of one group of 
stakeholders, possibly at the expense of another group. That 1s. no 
outcome measurement scheme can be free of values. and any evaluation of 
outcome must be in respect of a particular set of objectives (Smith, 1996, 


p29). 
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Bovaird agrees, stating, 


“Performance” is not a unitary concept, within an unambiguous meaning. 
Rather, it must be viewed as a set of information about achievements of 
varying significance to differing stakeholders. Performance measures 
must always be viewed within a multidimensional framework, and should 
be expected to be multidimensional in their form and content. If they 
appear one-dimensional, it is likely to be because they are of limited 
relevance. In practice, no performance information is other than partial, 
imperfect, and indicative (Bovaird, 1996, p. 147). 


A lack of consensus on program objectives is a common reality of the 
politically rational paradigm. When consensus does exist, it 1s a result of the political 
interaction among powerful stakeholders. The value of the paradigm lies in its 
recognition that public sector performance evaluation must first identify key stakeholders 
and their respective objectives before applying a technically rational measurement 
technique. The process of public sector performance measurement is thus broken-up into 


four stages: 


iY Identifying Stakeholders; 


2, Identifying Objectives: 
oe Developing Performance Indices; 
4. Applying technical rational performance measurement techniques. 


b. Identifying Stakeholders, Objectives, and Indices: The Objectives 
Hierarchy Approach 


A useful approach to systematically link key stakeholders and their 
respective objectives to performance indicators is the Objectives Hierarchy.4 An 
4 The Objectives Hierarchy is the first step in Thomas Saaty’s Analytical Hierarchy Process (AHP). While 


the researcher will use the Objectives Hierarchy approach throughout this study to break broad objectives 
down into sub-objectives and specific attributes, the remaining AHP techniques will not be applied. 
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Objectives Hierarchy stratifies into levels the relevant stakeholders and their respective 
objectives. A Hierarchy can be used to work down from each agency's high level 
objective (e.g., overall organizational performance) to relevant sub-objectives. to specific 
attributes that characterize those sub-objectives, and finally to measurable characteristics 
that allows quantification of the desired attributes (Melese. Lowe, and Stroup. 1997). A 
simple example of an objectives hierarchy for the Department of State is illustrated in 


Fieure |. 


International Stability 
Reduction in Conflicts Respect for Human Rights 
Around the World 


Prosecution of War Humane Prisons 
Crimes 


No Hostile Sustained 
Regional Development 
Coalition or Abroad 
Hegemon 
Exists 





KEY: 


Main Objective had Sub-Objective || Specific 
Attribute 


Figure |. Example of an Objectives Hierarchy 





In practice there is no set procedure for generating the objectives, sub- 
objectives, and specific attributes to be included in a hierarchy (Saaty, 1980). It is 
important to remember, however, that the ultimate organizational goal (objective) needs 
to be identified and placed at the top of the hierarchy. 

C. Summary 

Disagreement on evaluative frameworks exists not only because of 
technical obstacles, but also as a consequence of underlying political divisions (Fainstein, 
1987). Asa result, performance measurement can never be entirely scientific. Decision- 
making is much less systematic and rational than commonly supposed. As an aid to 
decision-making, performance evaluation is far less objective than the technically rational 
paradigm presumes. An understanding of the “governmental machine” of organizational 
processes and political actors 1s required before a measurement system can be developed 
(Allison, 1971). It is imperative to identify the stakeholders, and to what extent 
objectives consensus exists, prior to adopting any technically rational performance index. 
The Objectives Hierarchy process is a useful approach to analyzing multiple agencies and 


their objectives. 


C. CHAPTER SUMMARY 

Public sector performance evaluation 1s complex and controversial. Measurement 
of program efficiency involves collecting and reporting on input, throughput, and output 
factors in an effort to improve agency processes. Assessing the effectiveness of a 


program requires measuring outcomes against stated objectives. Various government 
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reports and private scholars have remarked on the lack of results-oriented measurement in 
government. Many reasons have been proposed for this. including institutional. 
technical, and financial obstacles. | Many practical reasons exist. however. for 
implementing performance measurement systems 1n government. 

A spectrum of approaches to public sector performance evaluation exists within 
the literature. Anchoring one end of the spectrum is the “technically rational paradigm” 
that presupposes objectives consensus and treats performance measurement as a coldly 
objective and scientific aid to decision-making. On the other end is the “politically 
rational paradigm” that recognizes that objectives consensus, when it exists. 1s a result of 
political bargaining among multiple stakeholders. It emphasizes identifving stakeholders 
and their respective objectives as a preliminary step in the performance measurement 


process. 
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Wl. METHODOLOGY 


A. INTRODUCTION 

This chapter will outline the methodology used to conduct this study. It will 
describe the selected research focus, the primary research methods used, and how the 
public sector performance evaluation process outlined in Chapter II is linked to the data 
collection and analysis stages of the study. The chapter will conclude with a detailed 
description of how each step in the public sector performance evaluation process was 


accomplished. 


B. RESEARCH FOCUS 

The objective of this thesis is to address how the Defense Security Cooperation 
Agency (DSCA) and the Departments of State and Defense can tackle the challenge of 
measuring the effectiveness and performance of their security assistance programs. The 
researcher chose to focus on the Expanded International Military Education and Training 
(E-IMET) military-to-civilian security assistance program due to the inherent subjective 
nature of international assistance efforts. A commonly held view within the security 
assistance community is that the effectiveness of security assistance programs cannot be 
measured with certainty due a lack of quantitative data. thereby making strict compliance 
with the 1993 Government Performance and Results Act impossible. It is’ the 
researcher s hypothesis, however. that objectively measuring the effectiveness of security 


assistance programs 1s possible. In order to place constructive limitations on the study, 


and because it is the potential core of an evolving military-to-civilian security assistance 
effort, the researcher focused entirely on the E-IMET program. 

In the spirit of the Results Act, any government agency and program could have 
been selected. DSCA was chosen as the central point of contact because it is organic to 
the Defense Department, is responsible for the implementation of the E-IMET program, 
and is faced with the same challenges as more traditional State Department security 


assistance organizations (1.e.,: The Agency for International Development (USAID)). 


C. RESEARCH METHODS 

This study makes use of two common public administration research methods. 
The first is the collection of data through unobtrusive measures, specifically archival 
research, and the second is the collection and verification of data through specialized 
interviewing. 

1. Unobtrusive Measures: Public Archival Data 

Webb et al. (1966) point out that archival sources of information have been 
widely used in public administration research for many years (Webb, Campbell, 
Schwartz, and Sechrest, 1966). Archival sources of data are written materials that are 
available from public and private records (Yaeger, 1989). Public archives include 
“ongoing records of society, such as actuarial records, voting records, government 
budgets, and communications media, which are produced and paid for by someone other 
than the researcher” (Yaeger, 1989, p. 714). The archival data collected in this study 


come solely from public records. 


Archival sources are further classified as primary or secondary. Primary public 
sources used in this study include speeches, congressional testimony. reports. and 
documents published by various U.S. government agencies. Examples include reports by 
the General Accounting Office (GAO) and agency-specific publications such as the 
National Military Strategy (DoD). Data collected from these primary public sources 
include formal organizational charts, missions, and objectives. 

Secondary public sources include the published books. articles, studies. papers. 
and lectures of academics (Yaeger, 1989). Secondary sources are used in this study to 
inform the performance measurement process and provide the background information 
necessary to cnitically analyze the data collected from primary public sources. An 
example of a secondary public source is the Public Administration Review, the oldest of 
the public administration journals (Yaeger, 1989). Data collected from these sources 
helped form the continuum of performance evaluation paradigms described in Chapter II] 
as well as the public sector performance measurement process used throughout this 
thesis. Secondary archival data also provided information critical to the development of 
the E-IMET performance indices found in Chapter V1. 

Archival sources are not without problems. however. First. archives may contain 
errors or be incomplete (Yaeger, 1989). Second. some archival materials “suffer from 
from selective deposit” (Yaeger. 1989, p. 720). Another common problem is that of 
selective survivability. This may happen with public sources when a_ political 
administration or agency removes records that reflect negatively upon the organization 


(Yaeger, 1989). While these problems might exist, the researcher took steps to Jimit their 
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effects. First, documents were checked for internal consistency. More importantly, 
multiple sources were used to ensure a fair picture emerged of a given issue. Agency- 
specific sources were compared to reports from other agencies, like the GAO, or to 
similar secondary sources. As will be described in the next section, key archival data 
were also verified through the specialized interview process. 

While there is no complete assurance that the data offer a comprehensive 
representation of all the stakeholders, the researcher made every effort to incorporate 
archival data from all parties. 

pa Specialized Interviews 

Interviews are frequently used to collect data and have been used throughout the 
history of public administration research (Yaeger, 1989). One type of interview, known 
as the elite or “specialized interview,” is used to collect data from a limited group of 
individuals or “key informants” (Yaeger, 1989, p. 698). Key informants are “individuals, 
who because of their position, are likely to have been involved either in making a 
decision of interest or in carrying it out” (Yaeger, 1989, p. 698). The special knowledge 
and experience that they have may enable them to comment on or verify the accuracy of 
the information gained during the course of the research project (Yaeger, 1989). With 
specialized interviews the number of subjects usually is limited. Rather than interviewing 
many subjects, key informants are selected because of the position they hold or becquse 
of their experiences. 

In August 1998 the researcher traveled to Washington D.C. to interview various 


key E-IMET informants. An example of a key informant used in this study is Ms. Rita 


Verry. head of the E-IMET program for DSCA. Specialized interviews with other 
selected officials within the E-IMET community were used to verify and help interpret 
various archival data. Data gathered during specialized interviews with various academic 
professionals at the Naval Postgraduate School (NPS), the Defense Resources 
Management Institute (DRMIJ), the International Health Resources Management (IHRM) 
Program. the Center for Civil-Mihtary Relations (CCMR), and the Defense Institute of 
International Legal Studies (DIILS) aided in the development of performance indicators 
for the E-IMET program. 

Various specialized interview formats exist. The one used in this study was the 
nonscheduled or nondirective interview. In this type of specialized interview the 
questions asked of each subject are not necessarily the same and may “differ in 
significant ways depending on the different positions and experiences of each subject” 
(Yaeger, 1989, p. 699). No preconceived set of questions was developed in advance in 
order to encourage the respondents to describe their experiences and interpretations as 
freely as possible. The questions posed during this study were often open-ended and 
intended to get the subject to talk in his or her own terms about the E-IMET program. 
The data collected through these interviews took shape as the process went along. with 
tentative generalizations being tested in subsequent interviews of other subjects. 

It is important to note that the key to conducting a specialized nondirective 
interview is sound background information. This is where the unobtrusive archival and 
the specialized interview research methods intersect to increase the adequacy. reliability. 


and representativeness of the data collected. Data gathered from archival material served 
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as the basis for the researcher’s questions during the interviews. The interviews served as 
a way of verifying the researcher's generalizations concerning performance evaluation of 
the E-IMET program. 

Like archival data, interview data are also not without errors. Potential problems 
include omission and underreporting, telescoping or overreporting, and lack of 
communication skills (Yaeger, 1989). These potential sources of error were overcome by 
the researcher through verification with other data sources and double-checking with the 
subject during the interview. Subsequent telephone interviews were also conducted when 


further clarification was needed. 


D. LINKING THE PUBLIC SECTOR PERFORMANCE MEASUREMENT 
PROCESS TO DATA COLLECTION AND ANALYSIS 


The process of public sector performance measurement used in this thesis is 


broken down into four iterative stages: 


l. Identify the Key E-IMET Stakeholders; 


2. Identify the Objectives of U.S. Security Assistance and the E-I[MET 
Program; 


Develop Performance Indices for the E-IMET Objectives; 
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4. Apply technically rational performance measurement techniques. 


The researcher will apply the first three steps of this process using data gathered 
from the two primary research techniques. Step #1 consists of data collection involving 
primary and secondary archival research. Step #2 continues the collection of archival 


data but concludes with the formation of a Global Objectives Hierarchy, thus moving the 


study into the analysis stage and triggering the specialized interview process. The 
analysis stage will conclude with a set of proposed performance indicators for each of the 
three E-IMET program objectives and is heavily dependent on data gathered through 
specialized interviews and secondary archival sources. Step #4. actually measuring the 
performance of the program, 1s beyond the scope of this study. Conclusions will then be 


drawn from the experiences and insight gained from the application of the process. 


E. THE PERFORMANCE MEASUREMENT PROCESS AND E-IMET 

1. Step #1: Identifying the Key E-IMET Stakeholders 

The purpose of this step is to identify and classify the key stakeholders of the E- 
IMET program. This process will take place in five sequential steps. 

First, the concept of a stakeholder will be clearly defined using secondary archival 
sources. Internal and external E-IMET stakeholders will then be identified using primary 
archival sources including security assistance doctrinal publications. Once individual 
stakeholders have been identified, they will be grouped according to their parent 
governmental agencies and the stakes they share in the evaluation of E-IMET 
performance. For example. DSCA, which ts identified as a key individual stakeholder. is 
grouped under its parent agency, the Department of Defense. The assumption here 1s that 
individual stakeholders within the same agency share common views on E-IMET 


performance evaluation and have similar effects on the E-IMET program? This 


> [It should be noted that it is entirely possible that individual stakeholders could have different views and 
effects on a given program than their respective agency as a whole. Within a government agency, 
however, it 1s reasonable to assume that the parent organization (DoD) would, to a certain extent, govern 
the effects of their organic individual agencies (DSCA), thus making a generalization (grouping) possible. 
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assumption is made to simplify the analysis in order to remain within the time constraints 
of the study. The basis for this grouping will be the organizational charts found in the 
primary archival sources. Each group's stake in E-IMET program will then be discussed 
in order to better understand the effects that they have on the program’s overall 
objectives. This 1s a very subjective step and will be informed by agency-specific 
documents and verified by subsequent specialized interviews. The last step in the 
stakeholder identification process is to formally organize the various stakeholder groups 
into a “stakeholder map” which will graphically present the potentially conflicting effects 
of the various stakeholders on E-IMET objectives consensus. This, too, is a subjective 
step, but will be informed by agency-specific documents and verified by subsequent 
specialized interviews. 


ue Step #2: Identifying the Objectives of U.S. Security Assistance and 
the E-IMET Program 


The purpose of this step is to construct a broad Global Security Assistance 
Objectives Hierarchy that will serve as a basis for objective performance evaluation and 
link the three E-IMET objectives back to each of the Executive Branch stakeholders.® 
This process will take place in five sequential steps. 

First, each stakeholder group’s security assistance objectives will be outlined 


using official agency-specific documents. Once outlined, a subjective sifting process will 


6 Only Executive Branch stakeholder objectives were outlined in this study. While Legislative Branch 
stakeholders are identified and have a profound effect on the E-IMET program, they do not set specific 
Administration security assistance direction. 


separate objectives from strategies (ends versus means) resulting in three separate 
agency-specific objective hierarchies. A Global Objectives Hierarchy will then be 
constructed using common Executive Branch stakeholder objectives. This Hierarchy will 
break broad national objectives down into sub-objectives and specific attributes that 
characterize those sub-objectives using the multiple objectives found in the existing 
individual hierarchies. While this process is inherently subjective it is more a function of 
pattern recognition among the separate stakeholder hierarchies. The next step is to 
validate the accuracy of the Global Hierarchy, as well as the objectives selection criteria 
used to construct it, through specialized interviews with key informants within the 
Defense Department and the State Department.’ An opinion was also sought from 
Legislative Branch key informants even though their objectives are not specifically 
incorporated into the Hierarchy. Once validated, the final step in the objectives 
identification process is to link the specific E-I[MET objectives to the Global Hierarchy. 
This last step forms the logical link between common national security assistance 
objectives and the E-IMET program. 

3. Step #3: Developing Performance Indices for the E-I[MET Objectives 

The next step in the public sector performance evaluation process, and the 
purpose of the last step in this study, is to propose indicators of performance for the three 


E-IMET program objectives. The three main objectives of the E-IMET program are: 


7 Representatives from the Executive Office of the President were unavailable for interview. Their 
national security assistance objectives are clearly defined in the National Security Strategy and the 
President's annual State of the Union Address, however. 
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1. Respect for Civilian Control Over the Military; 


2. Improved Military Justice Systems and Procedures In Accordance With 
International Human Rights; 


oe) 


Improved Defense Resource Management. 


This 1s the most critical and problematic step in the performance evaluation 
process. It involves identifying what qualities are inherent in each of the three objectives. 
Quantifiable indicators of performance must then be matched to the specific characteristic 
of the overall objective. In short, it requires a deep understanding of civil-military 
relations, military justice, and resource management. 

This step relies heavily on the data received from specialized interviews 
conducted with the academic professionals from the Naval Postgraduate School, the 
Defense Resources Management Institute, the International Health Resources 
Management Program, the Center for Civil-Military Relations, and Defense Institute of 
International Legal Studies. The experienced opinions of these professionals, combined 
with data gathered from secondary archival sources, will help form individual objectives 
hierarchies for each E-IMET objective. These hierarchies will then be used to develop 
indicators of performance for the E-IMET program. 

It is important to note that the result of the process is simply a starting point for 
further discussion. The indicators found in this study will be developed largely outside 
the influence of in-country security assistance officials. The researcher realizes that the 
inherently subjective nature of this process must be combined with informed country- 


specific data in order to yield the most beneficial measures of performance. The various 
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Security Assistance Officers (SAOs) who serve as the in-country point of contact for U.S. 
security assistance efforts were not interviewed for this study due to time and financial 
constraints. The performance indicators found in Chapter VI are a result of the critical 
analysis of the three broad E-IMET objectives and are meant as a starting point for the 


future development of country-specific performance indicators. 


F. CHAPTER SUMMARY 

1. The Public Sector Performance Evaluation Process 

The four-step method to measure the performance of public sector programs is an 
iterative process utilizing data gathered through archival and interview research. While 
many steps in the process are largely subjective, a concerted effort was made to 
crosscheck and verify each assumption. Primary archival data was compared to similar 
secondary archival data and specialized interviews were used to verify all assumptions 
and generalizations based on archival material. While every effort was made to gather 
data that 1s representative, adequate. and reliable, limitations did exist. 

ap Limitations of the Study 

The chief limitation of this study is the lack of past efforts to measure the 
effectiveness of security assistance programs. A standard evaluation format does not 
exist for security assistance programs. This created an enormous obstacle as the 
researcher was forced to break new ground concerning governmental program evaluation. 

Another limitation placed on the study was the lack of access to various E-IMET 


stakeholders. While some stakeholders were very accessible (DSCA and the Department 
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of State especially), many were not. The face-to-face interviews used in this study were 
conducted during a congressional recess making it difficult to adequately represent the 
Legislative Branch’s position. Representatives from the Executive Office of the 
President were also unavailable for interviews. Additionally, various in-country U.S. 
Embassy Security Assistance Officers were not contacted for their input concerning 


region-specific performance indicators. 


IV. DATA 


A. INTRODUCTION 

The purpose of the 1993 Government Performance and Results Act (GPRA) is “to 
improve Federal program effectiveness and public accountability by promoting a new 
focus on results, service quality, and customer satisfaction’ (GPRA, 1993. p. 2). GPRA 
mandated that each government agency submit to the Director of the Office of 
Management and Budget and to Congress a strategic plan for program activities (GPRA. 
1993). When developing a strategic plan, GPRA requires that the agency “consult with 
Congress, and solicit and consider the views and suggestions of those potentially affected 
by or interested in such a plan” (GPRA, 1993, p. 3). In short, GPRA requires each 
agency to work with their internal and external stakeholders to set agency direction. 
Thus, it is essential to consider the various stakeholders that affect the agency or program 
before a set of overall objectives can be developed. 

The purpose of this chapter is to identify the stakeholders of the E-IMET program. 
Once the stakeholders have been identified they will be grouped according to similar 
stakes and organized into an E-IMET stakeholder map. Chapter V will identufy the 
security assistance goals (objectives) of each stakeholder group based on their written 
documentation (1.e..: The Nationa] Security Strategy) in order to inform the E-IMET 
objectives setting process and enable the development of performance indicators for the 


E-IMET program. 


B. EXPANDED IMET STAKEHOLDERS 

i Identification of Stakeholders 

Measuring the performance of the E-IMET program must begin with the 
identification of key stakeholders. Freeman defines a stakeholder as “any group or 
individual who can affect or is affected by the achievement of an organization’s purpose” 
(Freeman, 1984). To identify key stakeholders Roberts and King ask a simple question: 
“What are the key stakeholder groups that are affected by or can affect the policies and 
programs which you are attempting to introduce?” (Roberts and King, 1989, p. 68). 

Descriptions of a stakeholder range from the highly specific to the general 
(Clarke, 1998). Stakeholders can be either internal or external to the organization or 
program (Roberts and King, 1989). They can also be potentially hostile or adversarial. 
Any group, internal or external. can potentially make a difference to the organization or 
program (Roberts and King, 1989). Once identified, individual stakeholders are grouped 
according to similar stakes and organized into an overall “stakeholder map” (Roberts and 
King, 1989). 

The presence of multiple stakeholder groups may result in diverse or conflicting 
ideas concerning a program’s objectives. Thus, the careful identification of dominant 
stakeholders is the critical first step in defining a single set of objectives for a program. 


Figure 2 contains a list of the dominant individual! stakeholders for the E-IMET program. 
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Senate Appropriations Committee for Foreign 


Operations (SACFO) 


Figure 2. Individual E-IMET Program Stakeholders 


a. Executive Office of the President Stakeholders 

The primary stakeholder organization within the Executive Office of 
President is the National Security Council (NSC). Chaired by the President. the 
function of the council is to advise the President with respect to the integration of 
domestic, foreign, and military policies relating to national security (DISAM, 1994). The 
NSC publishes the National Security Strategy of the United States that outlines the 
Administration s core national security objectives and strategic priorities. This document 
contains the Administration's broad goals and objectives relating to security assistance (.\ 
National Security Strategy for a New Century, 1997). It 1s the NSC, with guidance trom 


the President, that identifies core national security assistance objectives. 
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b. State Department Stakeholders 

The Secretary of State provides continuous supervision and general 
direction for security assistance, including military education and training programs 
(DSCA Website, 1998). The Department of State (DoS) affects program determination 
and integration. It is the DoS that determines whether there will be an E-IMET program 
for a country and, if so, its scope (DISAM, 1994). The DoS of is also responsible for 
“ensuring that the programs are effectively integrated both at home and abroad, and that 
the foreign policy of the United States is best served thereby” (DISAM, 1994, p. 79). To 
guide these decisions the Secretary of State develops international affairs strategic goals 
that are the foundation of the nation’s foreign policy (FY1999 Security Assistance 
Budget Request, 1998). 

The Under Secretary for International Security Affairs is the principle 
advisor to the Secretary of State and the focal point for security assistance matters within 
the Department (DISAM, 1994). The Under Secretary chairs the Security Assistance 
Program Review Working Group (SAPRWG) which is the DoS’s principle advisory 
body with policy planning and review functions. As an interagency group with wide 
representation throughout the Executive Branch, the SAPRWG acts as the central security 
assistance policy-making body for the Secretary of State (DISAM, 1994). 

The Bureau of Political-Military Affairs also plays a key security 
assistance role. It’s principle function is to advise the Secretary of State on issues and 


policy problems arising from areas where foreign policy and defense policy impinge on 
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one another (DISAM, 1994). The Bureau also plays a kev role in the administration of 
the IMET program. 
A diagram of the State Department’s organizational security assistance 


stakeholders 1s provided in Figure 3. 


Office of the Secretary of State 


Under Secretary for International Security Affairs 


SAPRW 


Bureau of Political-Military Affairs 


Figure 3. State Department Security Assistance Stakeholders 


Cc. Defense Department Stakeholders 

The Secretary of Defense is responsible for implementing programs to 
transfer security assistance services. The Defense Secretary is responsible for the 
performance of all defense functions that furnish military assistance. education. and 
training (DISAM, 1994). Department of Defense (DoD) strategic guidance is published 
in the National Military Strategy and the Quadrennial Defense Review (QDR). 

Authority and control over all DoD security assistance matters is delegated 
to the Under Secretary of Defense for Policy. The Assistant Secretary of Defense 
(International Security Affairs) (ASD/ISA), however, supervises security assistance 


programs with all foreign governments. Through the Defense Security Cooperation 
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Agency (DSCA), the ADS/ISA “interprets executive policy and develops DoD security 
assistance policies and programs” (DISAM, 1994. p. 84). 

The DSCA is established as a separate agency of the DoD under the 
direction and control of the Under Secretary of Defense for Policy and receives policy 
direction and supervision from ASD/ISA. DSCA handles many critical E-IMET 


functions including (DISAM, 1994); 


° Administration and supervision of E-IMET planning and training; 


° Coordination of the formulation and execution of the E-IMET program 
with other government agencies; 


. Developing and promulgating E-IMET procedures. 

Additional stakeholders exist within DoD. The Joint Staff organization 
determines the impact of security assistance programs on U.S. programs defense 
readiness, making it a key participant in the security assistance program development and 
review process (DISAM, 1994). The five Unified Commands are responsible for the 
conduct of the U.S. security assistance program within their respective regions (DISAM, 
1994). They provide critical program evaluation “of the efficiency and effectiveness of 
DoD overseas security assistance programs” (DISAM, 1994, p. 87). Each Military 
Department (MILDEP) maintains a separate department responsible for the actual 
delivery of security assistance services and provides recommendations to the Defense 
Secretary on the successful conduct of security assistance programs. 

A diagram of the Defense Department’s organizational security assistance 


stakeholders is provided in Figure 4. 
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Figure 4. Defense Department Security Assistance Stakeholders 


d. Legislative Branch Stakeholders 


Congress establishes the Jaws, authorizes the programs. appropriates the 


funds, and oversees the role that security assistance plays in international affairs. 


Specifically, the Congress has the power to (DISAM, 1994): 


4 Establish or amend basic security assistance authorization acts: 

° Enact security assistance appropriation acts: 

4 Pass Joint Resolutions to permit the incurrence of obligations to carry on 
essential security assistance program activities until appropriation action is 
complete; 

e Conduct hearings and investigations into special areas of interest. to 


include instructions to the General Accounting Office. Congressional 
Budget Office. and Congressional Research Office to accomplish special 


reviews: 


« Ratifv treaties that may have security assistance implications. 


The basic congressional stakeholder organization is the committee. 
Traditionally, powerful Authorization and Appropriation Committees, as well as their 
respective Subcommittees, are the dominant stakeholders. These committees provide 
program oversight during annual hearings on fiscal year foreign operations spending. 


The key Authorization Committees include (DISAM, 1994): 


e House [International Relations Committee (HIRC) 


e Senate Foreign Relations Committee (SFRC) 


The key Appropriation Subcommittees include (DISAM, 1994): 


. House Appropriations Subcommittee for Foreign Operations (HACFO) 
e Senate Appropriations Subcommittee on Foreign Operations (SACFO) 
om Stakeholder Issues, Concerns, or Stakes 


Once the key stakeholders have been identified it 1s necessary to determine the 


“stake” each has in E-IMET effectiveness evaluation. According to Roberts and King, 


A stake is the claim on the organization that each stakeholder has. The 
concept of a stake is based on the idea of one’s having something to lose 
Or gain in a given Situation, and therefore the nature of the stake depends 
on the issue at hand. Thus, the stake may be tangible (money, material, 
resources) or intangible (time, prestige, self-esteem), explicit or implicit. 
Moreover, a stake may be economic, political, social, or psychological in 
nature (Roberts and King, 1989, p. 66). 


a. The Grouping Process 
To simplify the analysis of stakes the individual stakeholders will 
immediately be placed into groups. Since the individual stakeholders are organic to 


higher level governmental agencies (i.e... DSCA is organic to the Department of 
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Defense) they can be grouped according to their formal organizational charts. It is 
assumed that since individual stakeholders are under operational and administrative 
control of their higher-level agency, they will possess a similar stake in the evaluation of 
the E-IMET program. It will then be possible to characterize the stakes of each group and 
construct an overall stakeholder map. The following stakeholder groups. along with their 


key individual stakeholders, have been identified for this study: 


The Executive Office of the President Group 


The President 
The National Security Council 


The State Department Group 


The Secretary of State 
The Under Secretary for International Security Affairs 
The Security Assistance Program Review Working Group 
The Bureau of Political-Military Affairs 


The Defense Department Group 


The Secretary of Defense 
The Under Secretary of Defense for Policy 
The Assistant Secretary of Defense (ISA) 
The Defense Security Cooperation Agency 
The Joint Staff 
The Unified Commands 
The Military Departments 


The Legislative Group 


The House International Relations Committee 
The Senate Foreign Relations Committee 
The House Appropriations Subcommittee for Foreign Operations 
The Senate Appropriations Subcommittee for Foreign Operations 


b. Stakes Analysis 
The following is a description of the stakes for each stakeholder group. It 
focuses on where each of the groups stands on the issue of E-IMET performance 
evaluation. 
(1) The Executive Office of President Group. The President 
outlines his national security priorities every year in the State of the Union Address. In 
1997 President Clinton cited the following security assistance priorities (National 


Security Strategy, 1997): 


° Foster a peaceful, undivided Europe; 

e Create a stable, prosperous Asia Pacific community; 

* Be a force for peace; 

e Maintain a strong military and superb diplomatic representation to meet all 


national security challenges. 


It is the responsibility of the National Security Council to develop 
the national security strategy, implementing the priorities of the nation’s Chief Executive. 
By setting national goals and objectives, the Council gives direction to DoS and DoD 
security assistance efforts. Thus, the Council has an implicit political stake in the 
effectiveness of security assistance programs like E-IMET. The degree to which a 
program like E-IMET helps accomplish presidential political priorities is of sigmficant 
interest to the President’s Executive Office. The NSC has a stake in measuring the E- 


IMET program’s ability to accomplish the President’s strategic priorities. 


(2) The State Department Group. The State Depart-ment 1s 
responsible for overseeing the development and integration of various security assistance 
programs throughout the Executive Branch. The Department submits the annual Security 
Assistance Budget Request that supports each of the U.S. foreign operations programs. It 
also publishes the annual Congressional Presentation for Foreign Operations (CPFO) 
that supports the funding requests for international security assistance programs. 
including the IMET program. Since current E-I[MET expenditures represent only a small 
portion of overall IMET funding, the State Department is concerned primarily with the 
broader IMET program (Bowab Interview, 1998). In 1998, the State Department claimed 
that the IMET program is an effective, low-cost component of the global U.S. security 


assistance effort (CPFO, 1999): 


The International Military Education and Training (IMET) program is an 
instrument of U.S. national security and foreign policy—a key component 
of U.S. security assistance. It is a program that, for a relatively modest 
investment, presents democratic alternatives to key foreign military and 
civilian leaders (CPFO, 1999). 


E-IMET funding, however. has been growing steadily as a 
percentage of IMET spending since the 1991 expansion of the IMET program. The 
State Department, therefore, has a tangible stake 1n assessing the effectiveness of the E- 
IMET program. A favorable E-IMET performance evaluation would demonstrate to 


Congress the cost effectiveness of the overall IMET program. A rigorous performance 


8 In 1998, Congress specified that approximately 30% of total IMET funding should be dedicated to the 
E-IMET program (Bowab, 1998). 
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evaluation that yields unfavorable results, however, would be problematic in terms of 
future congressional support. 

A favorable E-IMET performance evaluation also presents the 
State Department with a potential problem. In today’s environment of fiscal constraint a 
positive performance evaluation could also be used to justify the expansion of the E- 
IMET program at the expense of traditional IMET training. For those who believe that 
the current focus on E-IMET training is taking away from the State Department’s more 
traditional military-to-military IMET training, a positive review of the E-IMET program 
could cause an unwanted shift away from the IMET program. 

(3) The Defense Department Group. The Defense Depart- 
ment is responsible for the implementation of congressionally funded military security 
assistance programs. Specifically, the DSCA is tasked to provide the services associated 
with the E-IMET program. The Secretary of Defense’s Report of the Quadrennial 
Defense Review (QDR) and the Joint Staff's National Military Strategy outline a security 
strategy of engagement for the Defense Department and provide a mission for DSCA 
(QDR, 1997). DoD’s strategy of engagement emphasizes the role of security assistance 


in shaping the changing global security environment. 


A strategy of engagement presumes the United States will continue to 
exercise strong leadership in the international community, using all 
dimensions of its influence to shape the international security 
environment. This is particularly important to ensuring peace and stability 
in regions where the United States has vital or important interests and to 
broadening the community of free-market democracies. Strengthening 
and adapting alliances and coalitions that serve to protect shared interests 
and values are the most effective ways to accomplish these ends (QDR, 
10977). 
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Through peacetime engagement activities, U.S. Armed Forces promote 
regional stability, increase the security of allies and frends. build 
coalitions, and ensure a more secure global environment. The 
International Military Education and Training Program (IMET) 
establish[es] long-term professional relationships between our Armed 
Forces and the future military leadership of other countries (National 
Military Strategy, 1998). 


Similar to the State Department, the Defense Department has an 
explicit tangible stake in measuring the performance of DoD security assistance 
programs. Like the DoS, DoD 1s subject to GPRA. The requirement to measure the 
performance of security assistance programs like E-IMET presents DoD with a double- 
edged sword. A favorable measurement of effectiveness might guarantee future funding. 
but a negative report could jeopardize the program. 

The Department of Defense also has an intangible stake in 
measuring E-IMET program effectiveness. As the international security environment 
develops DoD struggles to clearly identify its changing mission. With its growing role in 
non-traditional military functions, such as humanitarian assistance and peacekeeping 
operations, a key future DoD mission might be the delivery of international military-to- 
military and military-to-civilian training. One might conclude that DoD’s future 
relevance partially rests on ensuring the continued support for military security assistance 
programs like E-IMET. A favorable E-IMET performance evaluation would go a long 
way towards assuring continued congressional support for DoD security assistance 


programs. 
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(4) The Legislative Group. In 1993 President Clinton signed 
into law the Government Performance and Results Act. Also in 1993, the President and 
Vice President initiated the National Performance Review (NPR) to reinvent government 
(NPR Benchmarking Report, 1997). As Hayward points out, however, “GPRA is more 
than an administration directive that can be rescinded or whose implementation can be 
stalled while waiting for a new administration with a different agenda” (Hayward, 1998, 
p. 2). In fact, Congress aspires to completely enhance government performance and 
results (Hayward, 1998). Representative Deborah Pryce of Ohio states very clearly the 


stake the Legislative Branch has in government performance evaluation: 


As the custodians of our nation’s purse strings, Congress has an incredible 
responsibility. We have been entrusted to safeguard the hard-earned 
money that the taxpayers send to Washington. It is our responsibility to 
see to it that those dollars are spent wisely and that the American people 
get the biggest bang for this buck (freedom.house.gov/results Website, 
1998). 


Many other Congressional Senators and Representatives share 
similar feelings (see Armey, Burton, Horn, and Kingston, freedom.house.gov/results 
Website, 1998). In fact, the House Majority Leader has an Internet Website supporting 
and tracking the progress of the Results Act (http://freedom.house.gov/results). 
Improving the public confidence in the federal government by systematically holding 
federal agencies accountable for achieving program results has received broad support 
within the Legislative Branch. 

Congressional support for GPRA represents an explicit intangible 


stake in E-IMET performance evaluation. Congress has clearly voiced their support for 
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legislation that holds agencies accountable for results. Failure to follow through on their 
promise would be a serious blow to the esteem and prestige of the people’s elected 
officials. Regularly elected government officials have an incentive to see that the 
constituency they represent receives value for their tax dollars. 

(5) Summary. Four different groups of stakeholders have a 
stake in E-IMET performance evaluation. The Executive Office of the President has a 
political stake in determining whether the E-IMET program meets the President's 
national security priorities. The Departments of State and Defense are each committed to 
the role of security assistance in the changing global security environment. [Each 
department is also subject to the 1993 Government Performance and Results Act. Both 
departments have a stake in demonstrating the effectiveness E-IMET in order to ensure 
the program's continued existence. While the State Department tends to focus on the 
effectiveness of the broader IMET program, the Defense Department recognizes E-[MET 
as a separate program requiring its own set of performance indicators. The Legislature 
has a stake in the government-wide adherence to the 1993 Government Performance and 
Results Act. The people's elected representatives want to prove to their constituents that 


their money is not being wasted in Washington. 


c. THE STAKEHOLDER MAP 
Based on their different stakes in the evaluation of the E-I[MET program the 
various stakeholder groups can finally be organized into an overall stakeholder map. 


Figure 5 contains the E-I[MET stakeholder map. 
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Figure 5. The E-IMET Stakeholder Map 


D. CHAPTER SUMMARY 

The development and management of the E-[MET program involves the active 
participation and cooperation of several key stakeholders within the national government. 
The National Security Council of the President’s Executive Office establishes the 


National Security Strategy that guides all other Executive Branch security assistance 
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efforts. By law, the Secretary of State is responsible for the supervision and general 
direction of all U.S. security assistance efforts. The Department of Defense is responsible 
for implementing military programs such as E-IMET. Through the DSCA. the DoD “has 
the largest supportive role from a level-of-effort standpoint” and is the most critical E- 
IMET stakeholder (DISAM, 1994. p. 95). 

E-IMET stakeholders also exist in the Legislative Branch of the national 
government. Through its oversight role Congress authorizes and appropriates public 
funds necessary to keep the program alive. In an effort to ensure responsible spending 
Congress has demanded performance-based budgeting on the part of all government 
agencies. 

Because it 1s assumed that individual stakeholders within the same higher-level 
governmental agency share common stakes in E-IMET performance evaluation, a 
erouping of stakeholders according to similar stakes is possible. Figure 5 represents the 
stakeholder map for the E-IMET program. Figure 6 summarizes the flow of 
communications for security assistance issues from the perspective of the Defense 


Security Cooperation Agency (DSCA Presentation, 1998). 
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Figure 6. Security Assistance Communications (DoD Perspective) 
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Ne ANALYSIS: PART I 


A. INTRODUCTION 

The Government Performance and Results Act of 1993 states that “federal 
managers are seriously disadvantaged in their efforts to improve program efficiency and 
effectiveness, because of insufficient articulation of program goals and inadequate 
information on program performance” (GPRA, 1993, p. 2). GPRA emphasizes the 
setting of clear program goals and the measurement of program performance against 
those goals (GPRA, 1993). Developing a clear set of agreed-upon program goals 
(objectives) is often problematic, however. As Chapter IV clearly demonstrates, the E- 
IMET program has many internal and external stakeholders. It is possible that each 
stakeholder defines the program's objectives differently. If a common definition of 
program objectives does not exist, than any attempt to objectively measure program 
performance will ultimately fail as multiple constituencies quarrel over whose set of 
objectives will serve as the performance baseline. Thus, it is necessary to analyze 
individual stakeholder objectives in order to determine if a single set of program 
objectives exist. 

The purpose of this chapter is to identify each stakeholder’s security assistance 
objectives to determine if a single set of common objectives emerge. Using written 
documentation from each group of stakeholders. security assistance objectives were 
singled-out and compared to other stakeholder groups. Ultimately, a single set of 


national security assistance objectives emerged and was arranged into a “Global 
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Hierarchy.” Specific attributes were then identified for each of the common global sub- 
objectives and used to link the E-IMET program objectives back to the Global Hierarchy, 


creating an objective baseline for future performance measurement efforts. 


B. THE GLOBAL HIERARCHY 


1. Methodology 


The bulk of this chapter will present the logic used to construct the Global 
Hierarchy. It will outline the steps in the process, highlight the key assumptions that 
were necessary, and examine the difficulties encountered during the process. The chapter 
will culminate with a Global Objectives Hierarchy for security assistance. 

a. Step #1: Document Review 

In order to develop a Global Objectives Hierarchy for security assistance 
One must start with a detailed review of the key stakeholder documents that contain the 
stakeholder’s security assistance objectives. A basic assumption is that, since the role of 
the Executive Branch of the national government is to set policy, only Executive Branch 
stakeholder documents should be used to form the Global Hierarchy. The following 


Executive Branch stakeholder documents were analyzed: 


Executive Office of the President: 


1997 National Security Strategy 
1997 State of the Union Address 


Department of State: 


Fiscal Year 1999 Security Assistance Budget Request 
1998 Department of State Strategic Plan for International Affairs 
1998 Congressional Presentation for Foreign Operations 
Fiscal Year 1999 Department of State Performance Plan 
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Defense Department: 


Fiscal Year 1999 Support for the Security Assistance Budget Request 
1997 Quadrennial Defense Review 
National Military Strategy 
Fiscal Year 1999 Department of Defense Performance Plan 


The preceding list of documents is not exhaustive. It 1s, however. a fair 
representation of the key documents that guide Executive Branch stakeholder security 
assistance policy. In addition, as with the document selection process. all remaining steps 
used to construct the Global Hierarchy concentrates only on Executive Branch 
stakeholders. 

b. Step #2: The Sifting Process 

Once the key stakeholder documents were chosen a sifting process 
Separating security assistance objectives from other unrelated objectives followed for 
each stakeholder group. This process is largely subjective, although many of the 
documents clearly identified objectives as being security assistance related. Great care 
was taken to include all specific security assistance objectives as well as those that have 
only a marginal relation to the stakeholder’s security assistance role. Ultimately. a 
separate objectives hierarchy was constructed for each stakeholder group using the sifting 
process and the selected documents. 

The result was three stakeholder group hierarchies from which to form one 
Global Hierarchy. The purpose of first constructing three individual hierarchies was to 
allow for a comparison of security assistance objectives among the stakeholder groups. 


Objectives consensus is the foundation for the Global Hierarchy and the kev prerequisite 
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for objective performance evaluation of the E-IMET program. By comparing the 
hierarchies to one another it is possible to determine the degree of consensus among the 
stakeholders’ objectives. 

(1) Sifting Process Challenges. Extracting specific security 
assistance objectives from the key stakeholder documents was a difficult and highly 
subjective process. Four specific difficulties were encountered during the sifting process. 

(a) Separating Objectives from Alternatives: Ends versus 
Means. A common problem found in the key stakeholder documents was the mixing of 
objectives with specific strategies available to accomplish those objectives. Few federal 
documents systematically break down broad objectives into their relevant sub-objectives. 
Instead, many of the stakeholder documents highlighted a broad objective (“ends”) and 
proceeded directly into a discussion of the alternatives available to achieve the final result 
desired (“means”). An example can be found in the Defense Department’s FY99 
Performance Plan. Promote Stability, which is an ends or objective, is immediately 
followed by Overseas Presence, which is a means or method for accomplishing the 
objective (DoD Performance Plan, 1998). This confusion inevitably results in a lack of 
measurable detail concerning the overall objective that is required if one 1s to determine 
whether the program achieved its objectives. In other words, while government agencies 
are busy identifying methods to achieve an objective, they fail to sufficiently expand 
enough on their objectives to make measurement possible. 

(b) Separating Security Assistance Objectives from 


Unrelated Objectives. Extracting from the key stakeholder documents the objectives and 
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sub-objectives that relate only to security assistance was the most difficult part of 
constructing the Global Hierarchy. Each document contained references to stakeholder 
objectives that are not specifically related to their security assistance mission. An 
example can be found in the Department of State’s FY99 Performance Plan. The State 
Department lists Stabilize World Population Growth as a global objective. Clearly this 
objective 1s not related to the definition of security assistance from Chapter I. and should 
not be included in a global security assistance hierarchy. 

Again, this is a highly subjective process as many 
sovernmental objectives are interdependent. It is essential. however, to separate specific 
security assistance goals from unrelated goals in order to construct a single hierarchy. 

(c) Practice in Constructing a Global Hierarchy. As Saaty 
points out, in practice there 1s no set procedure for generating the objectives and sub- 
objectives included in a hierarchy (Saaty, 1980). In fact, constructing a global security 
assistance hierarchy proved to be an original exercise. No individual interviewed from 
any of the Executive or Legislative Branch stakeholder agencies knew of any previous 
effort to link national security assistance objectives using a Global Hierarchy. The lack 
of practice in creating a Global Objectives Hierarchy resulted in a significant challenge to 
the overall process. 

(a) Time Commitment. One explanation for the lack of 
practice in constructing a Global Hierarchy is the amount of time it takes to analyze the 
key documents for each E-IMET stakeholder. Since responsibility for various parts of 
the E-IMET program is shared among three primary agencies within the Executive 
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Branch, the process of constructing individual hierarchies and synthesizing them into 
one Global Hierarchy was very time consuming. 

Cc Step #3: Constructing the Global Hierarchy 

The next step is to actually construct the Global Hierarchy. The existence 
of objective consensus among the individual stakeholders is vital to the construction of an 
objectives hierarchy. Fortunately, once ends were separated from their means, there 
emerged a consensus of basic security assistance sub-objectives among Executive Branch 
stakeholders. 

Each stakeholder defined their individual objectives in relation to The 
President’s three core national security objectives of enhanced security. economic 
prosperity, and a democratic world (National Security Strategy and State of the Union 
Address, 1997) (See Figure 7). Further, agreement existed among the remaining 
stakeholders concerning their definition of the sub-objectives of the President’s core 
goals. These common sub-objectives include international stability, uninhibited 
access to key markets, energy supplies, and strategic resources, and support for 
democratic principles (See Figure 7). 

The next step in the construction process was to further break down each 
broad sub-objective into the specific attributes that characterize that sub-objective. Here 
the stakeholders defined the specific attributes from their point of view. For instance, the 
State Department recognized that uninhibited access to key markets, energy supplies, and 
strategic resources require the growth of transitional and developing economies from 
socialist to market-based economies. The Defense Department, however, recognized that 
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the same sub-objective requires access to foreign military bases. Are one right and the 
other wrong? Within the context of the Global Hierarchy. the mission of national 
security assistance should include both stakeholder attributes. Building the Global 
Hierarchy, therefore. 1s a process of incorporating each stakeholder’s relevant attributes 
into the larger picture. This is exactly as GPRA 1993 dictates—the inclusion of internal 
and external stakeholders into the objectives-setting process. The Global Hierarchy. 
therefore, starts with a common set of broad sub-objectives and incorporates each 
stakeholder’s relevant attributes. 

The final Global Objectives Hierarchy can be found in Figure 7. 

d, Step #4: Validation 

Ultimately, the resulting Global Hierarchy, and the criteria used to select 
the sub-objectives and specific attributes. was validated through an interview process 
with various Executive Branch stakeholders. The validation process consisted of 
interviews with officials from the Departments of State and Defense.? Officials were 
chosen from the organizations within each stakeholder group that deal most often with E- 
IMET issues. They are assumed to represent the views of the entire stakeholder group. 


Interviews were conducted with officials from the following organizations: 


Department of State: 


The Bureau of Political-Muilitary Affairs 


9 Officials from the Executive Office of the President were unavailable for interviews. The 
National Security Strategy and the President’s State of the Union Address. which define the ultimate 
objectives placed on top of the Global Iherarchy, clearly convey the views of that key stakeholder. 
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Executive Office of the President (EoP): 
I, 1997 National Security Strategy 
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Department of State (DoS): 
I. FY99 Security Assistance Budget Request 


2. 1998 Strategic Plan for International Affairs 
3. 1998 Congressional Presentation for Foreign Operations 
4, FY99 Performance Plan 


Department of Defense (DoD): 
. Support for the FY99 Security Assistance Budget Request 


I 
z. 1997 Quadrennial Defense Review 
3. 1997 National Military Strategy 

4. FY99 Performance Plan 
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(1) Validating the Objectives Selection Criteria. Two 
simple criteria were used to select the individual stakeholder objectives and specific 
attributes that would be incorporated into the Global Hierarchy. 

The first criterion was that the stated objective had to fit the 
definition of an “end.” In other words, to qualify as an objective it could not be 
interpreted as “means,” or an alternative. It must be a clear statement of the final result 
desired, not a method to get there. 

The second criterion was that the objective had to relate to security 
assistance, either directly or indirectly. This criterion 1s purposefully subjective and 
broad in recognition of the fact that security assistance takes many forms and js related to 
many objectives. 

Each stakeholder interviewed validated the criteria used to select 
objectives for the Global Hierarchy. 


(2) Validating the Global Hierarehy. Each stakeholder 





group interviewed also validated the final Global Hierarchy. No disagreement existed 
among the stakeholders as to the relationship between the highest level objectives defined 
by the President, the common sub-objectives, and the specific attributes that characterize 


the sub-objectives. 
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It should be stressed that, while the Global Hierarchy was validated 
through numerous stakeholder interviews, national security assistance objectives will 
likely change over time. As such, the Global Hierarchy will require periodic revalidation 
as international security conditions evolve. Revalidation efforts should also include new 
stakeholders whenever possible. 


2. Linking the E-IMET Objectives to the Global Hierarchy 


Once the Global Hierarchy was constructed and specific attributes were identified 
the next logical step was to link the specific E-IMET program objectives back to the 
hierarchy. If an E-IMET objective could be logically linked to the achievement of a 
specific attribute within the Global Hierarchy a solid tnangle, circle. or box was placed 
under the attribute. This linkage is made possible by the development of individual 
hierarchies for each E-IMET objective. For example, improved military justice systems 
in accordance with international human rights can be linked to the existence of 
neutral law enforcement within a country since neutral law enforcement bodies are 
central to Western-style military justice systems (See Figure 7). 

Because performance measurement systems do not yet exist within the security 
assistance community, any attempt to link E-IMET objectives back to the Global 
Objectives Hierarchy is largely subjective. Quantitative data have yet to be collected that 
objectively conclude that E-IMET programs have resulted in the achievement of higher 
global objectives. The development of individual hierarchies for each E-IMET objective, 
however, provides a formal logic for the linkage. Additionally, anecdotal evidence, in the 


form of congressional testimony from various State and Defense Department officials, as 
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well as stakeholder validation. is available. however.!9 Furthermore. interviews with the 
Executive Branch stakeholders validated the logical link between the Global Hierarchy 
and specific E-IMET objectives. 

The validated Global Hierarchy and its connection to the specific E-IMET 


objectives can be found in Figure 7. 


Cc CHAPTER SUMMARY 

Figure 7 contains the Global Objectives Hierarchy for security assistance. To 
construct the hierarchy a sifting process was applied to key Executive Branch stakeholder 
documents in an effort to identify their individual security assistance sub-objectives and 
specific attributes. While the President defines the ultimate security assistance objective. 
a common set of sub-objectives was identified for each stakeholder group. To complete 
the hierarchy each stakeholder’s specific attributes for the common sub-objectives were 
added. Thus, in the spirit of the Government Performance and Results Act of 1993. a 
Global Hierarchy emerged that incorporates each stakeholder group into the objectives- 
setting process. 

Figure 7 also shows the logical link between specific E-IMET objectives and the 
national security assistance objectives. While no quantitative performance data exists to 
objectively link specific E-IMET objectives back to the Global Hierarchy, anecdotal 


evidence is available. Now that specific E-IMET objectives have been logically linked to 


10 Anecdotal evidence concludes that IMET and E-IMET programs are an integral part of the national! 
security assistance effort. Examples include congressional testimony from military officers, including 
various regional Commanders-in-Chief (CINCs), and various DoD and DoS officials (see Larson 1993. 
McCaffrey 1995, Wilhelm 1998, Slocombe 1998, Shattuck 1996, and Hamilton, 1997). 
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national security assistance objectives the challenge is to develop performance indices 


that objectively measure the effectiveness of the E-IMET intervention. 
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VI. ANALYSIS: PART II 


A. INTRODUCTION 
The third step in the public sector performance evaluation process is the 
development of performance indices for each program objective. The objectives of the E- 


IMET program are: 


1. Respect for Civilian Control Over the Military: 


2. Improve Military Justice Systems and Procedures In Accordance With 
International Human Rights; 
3. Improve Defense Resources Management. 


In order to evaluate whether various E-[MET interventions are having the desired 
effect one must measure results against the origina! objectives. This chapter will identify 
specific performance indicators that can be used to evaluate the effectiveness of the E- 
IMET program and highlight the key difficulties encountered in the process. 

The indices proposed in this chapter stem from the data collected through 
secondary archival research and specialized interviews. It is important to note that the 
indices developed here are general in nature and have not been patterned to a specific 
country or region. This chapter 1s not meant to be an exhaustive list of E-IMET 


performance indices, but rather a starting point for future development. 


B. THE IMPORTANCE OF A BASELINE ASSESSMENT 
The performance evaluation process is essentially a comparison of  post- 


intervention results against objective data gathered during a baseline assessment. A 


baseline assessment is defined as the “initial collection of data to establish a basis for 
comparison” (National Performance Review, 1997, p. 27). The success of the E-IMET 
performance evaluation process hinges on the objective initial assessment of a nation’s 
civilian leadership, military justice system, and ability to efficiently manage defense 
resources. If one does not possess a clear picture of civil-military relations, military 
justice, and resources management within a country before the initial intervention, 
attributing future incremental changes to the E-IMET program is not possible. All 
program results must therefore be stated in terms of a progression away from the baseline 
assessment. 

The performance indices proposed in this chapter assume that an objective 
baseline assessment is conducted by U.S. Embassy staff (1.e.: The Security Assistance 
Officer (SAQO)) or other DoD officials before a country is selected to receive E-IMET 
training. Currently, “needs assessments’ are routinely conducted as part of the first phase 
of many E-IMET programs. It is unclear, however, whether current needs assessments 
are intended to form the objective baseline necessary for incremental performance 
evaluation or simply an effort to match specific E-IMET programs to perceived needs 
within a country. The design and implementation of an E-[MET performance evaluation 
baseline assessment is beyond the scope of this study, but its importance within the 


performance evaluation process should not be overlooked. 
OF INDICATORS OF PERFORMANCE FOR THE E-IMET PROGRAM 

1. Civilian Control Over the Military 

The first E-IMET objective is the promotion of understanding and respect for 
civilian contro] over the military. In order to propose performance indicators for this 
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objective one must define the specifics of civilian contro] over the military. 
Unfortunately, many scholars debate the characteristics of civilian control over the 
military in terms of modern civil-military relations. !! 

a. Civilian Control versus Good Civil-Military Relations 

The congressional legislation authorizing the expansion of IMET training 
to include civilian contro] over the military concentrates only on the act of civilian 
control, not on the dynamics of good civil-military relations (U.S. Congress. 1991). The 
difference is subtle, but enormously important in terms of performance evaluation. 
Evaluating the quality of a country's civil-military relations is largely subjective, but 
evaluating the degree of civilian control over the military is fairly clear cut. While 
scholars disagree on what conditions are best for good civil-military relations. there is 
rough agreement on what constitutes civilian control. During the Cold War both the 
United States and the Soviet Union enjoyed civilian dominance over their expansive 
militaries, but scholars often characterize U.S. and Soviet civil-military relations quite 
differently (see Michael C. Desch’s “Threat Environments and Military Missions.” in 
Diamond and Plattner’s Civil-Military Relations and Democracy, 1996). E-IMET 
performance indices, therefore, must concentrate only on the changes within a country 
that results in greater civilian contro] over the military establishment. Whether gains 
toward civilian control result in better overall civil-military relations and a more peaceful 


or democratic society is another question entirely. 


I! The writings of Huntington 1996, Desch 1996 and 1998, Avant 1998, Bracken 1995, and Segal 1995 
offer lively debates concerning civilian control over the military and modern civil-military relations. 


is 


b. Civilian Control Objectives Hierarchy and Performance Indices 

In order to develop performance indices for civilian control over the 
military, one must first break down the broad objective of civilian contro] into multiple 
sub-objectives and specific attributes. The goal of this process 1s to construct a civilian 
control over the military objectives hierarchy that can be used to identify subsequent 
performance indicators. Drawing heavily from the U.S. model of civilian military 
control, and following the same format as the Global Security Assistance Hierarchy 
found in Chapter V, the objective of civilian control over the military can be broken down 
into the following sub-objectives: 

Sub-Objective #1: 
Military’s Organizational Structure and Culture Emphasize Civilian Dominance 
Sub-Objective #2: 
The Military is used to Counter External Threats vice Internal Problems 
Sub-Objective #3: 
Presentation of the Military is Through an Objective Source! 

Specific attributes that characterize each sub-objective must then be 
identified in order to complete the hierarchy. Based on the specific attribute, it 1s possible 
to propose indicators of performance that can be used to report whether the E-IMET 
intervention is achieving its stated objective. The specific attributes of each sub- 


objective, along with their respective indicators of performance, follow. 


12 This sub-objective refers to the right of the citizenry to receive information concerning the activities and 
policies of their military without state-sponsored censure. 
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Sub-Objective #1: 


Military’s Organizational Structure and Culture Emphasize Civilian Dominance 


N 


Specific Attributes: 


The Commander-in-Chief of the armed forces 1s a civilian and shares 
power with a popularly elected legislature. 


Indicators of Performance: 


e The Commander-in-Chief commissions officers after first nomina- 
ting them to the people's popularly elected representatives. 


e The Commander-in-Chief promotes officers after first nominating 
them to the people's popularly elected representatives. 


e The Commander-in-Chief 1s a civilian. 


Norms exist within the military for subordination to civilian control. 


Note: This attribute can be further divided into the following two 
attributes: 


a. The military is politically neutral. 


Indicators of Performance: 


‘ The military accepts all missions laid out by the civilian 
authorities. 
e The military does not try to influence policy-making 


(specific limitations and constraints are placed on the 
military’s ability to influence policy debates). 


e Civilians decide on ends of policy while the military 
decides on the means. and civilian authorities define the 
line between ends and means. 


° The military 1s professionally vice ideologically driven. 


* The military does not impose its values upon civilian 
Institutions and organizations. 
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>) 


e Special military bodies exercising political power are 
eliminated. 


b. A dialogue exists between civilian and military experts. 


Indicators of Performance: 


2 Open debate exists within military and civilian commun- 
ities. 
. Civilian and military staffs work together to solve long 


term problems and issues. 


~ The number of major conflicts between civilian and 
military leaders decreases. 


4 The public has access to military leaders through their 
elected representatives. 


The civilian leadership maintains total legal control over the military. 


Note: This attribute can be further divided into the following two 
attributes: 


a. The military’s budget is controlled by the popularly elected 
legislature. 


Indicators of Performance: 


o The legislature approves the military’s budget and 
subsequent expenditures (“The Power of the Purse’’). 


e The legislature approves the final composition of all 
military bases and depots. 


- The overall size of the military is controlled by the civilian 
legislature. 
a The legislature approves the composition of forces. 
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b. The civilian-military chain of command is legally based. 


Indicator of Performance: 


e A formal civilian-controlled chain of command exists and 
is based in law. 


Civil-military organizational relationships emphasize civilian control. 


Note: This attribute can be further divided into the following two 


attributes: 
a. The civilianization of high political offices. 
[Indicator of Performance: 
. Movement of civilians into traditional military positions 
within the Defense Ministry. 
b. No excessive influence by the military of the military-industrial 
complex. 
Indicators of Performance: 
° The legislature controls the military’s relationship with 
industry. 
° Military-run business and industries are privatized. 


Sub-Objective #2: 
The Military is used to Counter External Threats vice Internal Problems 
Specific Attributes: 
Externally oriented military mission and doctrine. 
Indicators of Performance: 


. Military doctrines are implemented with an external focus. 
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tO 


e The curricula of service academies are revised stressing civilian 
control and external missions for the military. 


. Laws regulating the use of the military for internal purposes are 
implemented. 

e The military redeploys forces away from internal missions. 

e The military defends society against external threats. 


Internal problems are addressed without the use of the military. 


Indicator of Performance: 


e Internal problems, formerly addressed through military means, are 
now addressed with non-military state agencies. 


Sub-Objective #3: 
Presentation of the Military is Through an Objective Source 
Specific Attributes: 


A free domestic media. 


Indicators of Performance: 


e Open dialogue exists between the press and the military. 
+ Governmental control (censorship) of press reporting 1s 
discontinued. 


Free public access to published government information. 


Indicators of Performance: 


© The public has free access to military budget data and deployment 
Statistics. 
. The public has free access to the voting records of elected officials. 
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The complete hierarchy of the sub-objectives and specific attributes for the 


civilian contro] objective can be found in Figure 8. 
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Gc Limitations 

While the researcher believes that objectively measuring the progress 
toward civilian contro! over the military is possible, limitations do exist. First, many of 
the performance indicators measure only structural changes in the military organization. 
Cultural and behavioral changes are often more important and much more difficult to 
measure. This limitation should not overshadow, however, the importance of detecting 
structural changes, as they are necessary in order to bring about the changes in behavior 
that characterize the culture of civilian control. 

A second limitation involves the long-term nature of organizational 
change. Segal warns that that civilian control and good civil-military relations is a 
process, not an existential state (Segal, 1995). Attempts to measure civilian control over 
the military should not be single snapshots in time but rather a series of snapshots over 
the duration of the program. Each measure must be continually compared to the baseline 
assessment in order to determine if the E-[MET intervention had the intended effect. 

Another important limitation is the inability to control external variables 
when measuring the effectiveness of the E-IMET intervention. Huntington points out 
that one reason why so many military regimes in Latin America have voluntarily 
surrendered power is due not to the efforts of the U.S., but rather to the recognition on the 
part of military leaders “that there are no easy solutions to the intractable economic, 
social, and political: problems confronting their respective countnes, and that the 
sustained involvement in politics has disastrous effects on the coherence, efficiency, and 
discipline of the army” (Huntington, 1995, p. 7). Military leaders selfishly chose to avoid 
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politics in an effort to regain control over their armies and leave the tough decisions to 
others. In this case. survival, rather than a dedication to the tdeal of civilian control. 
motivates the military’s surrender of power. Situations such as this make it difficult to 
separate the effects of the E-[MET intervention from the external environment. 
Contentious indicators of civilian control over the military also exist that 
make performance evaluation more complicated. Scholars disagree. for example. 
whether military representativeness of society 1s an indicator of civilian control. 
Huntington believes that a professional and politically neutral military is an indicator of 
civilian control and proper civil-military relations, regardless of its societal representation 
(Huntington, 1995). Janowitz, on the other hand, believes that proper civilian control 
starts with the “citizen-soldier” and a military that mirrors the society it defends (Segal. 
1995). For this reason, the researcher chose not to include military representativeness of 
society as an indicator of performance. Other contentious indicators, such as the 
abolition of conscription, were also intentionally left out because it is unclear that they 
indicate greater civilian control over the military. 
a. Improved Military Justice Systems 
The 1991 law that expanded [MET training specifically called for new courses 
focused on “creating and implementing effective military justice systems and codes of 
conduct with a special emphasis on the protection of human rights’ (U.S. Congress. 1991. 
p. 135). The new law called for a dedicated training effort aimed at improving the 
military justice systems of current and potential U.S. allies in accordance with 
international human rights standards. As with civilian control over the military. assessing 
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the progress toward this objective is a matter of measuring post-training gains toward 
more humane military justice systems. 


a. Improved Military Justice Systems Objectives Hierarchy and 
Performance Indices 


In order to propose performance indicators for this objective one must 
define the essential characteristics and objectives of a humane system of military justice. 
The U.S. system of military justice will serve as the model for this study. Using the 
objectives hierarchy approach the broad objective of improved military justice systems 
will be broken down into multiple sub-objectives and specific attributes patterned after 
the U.S. model of military justice. The goal of this process is to construct an improved 
military justice systems objectives hierarchy that can be used to identify subsequent 
performance indicators. The improved military justice systems objective can be broken 


down into the following broad sub-objectives: 


Sub-Objective #1: 
Eliminate the Potential for War Crimes 
Sub-Objective #2: 
Eliminate Peacetime Human Rights Abuses by the Military 
The second sub-objective can be further broken down into the following 
two additional objectives: 
Additional Objective A: 


Establish a Western Style Military Justice System That Recognizes the 
Rights of the Individual 
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Additional Objective B: 


Civilian Control Over the Military 


In this case sub-objective #1 shares common attributes with both the 
additional objectives identified for sub-objective #2.!3 The following common specific 
attributes, along with their respective indicators of performance, have been identified for 


the additional objectives of sub-objective #2: 


Additional Objective A. 


Establish a Western Style Military Justice System That Recognizes the Rights of 
the Individual 


Specific Attributes: 


ie Established legal system based on the assumption of innocence. the law of 
evidence, and the burden of proof. 


Indicators of Performance: 


e A new or revised military justice code 1s adopted that contains 
written rules akin to the U.S. Uniform Code of Military Justice 
(UCMJ) which define offenses, authority and jurisdiction require- 
ments, and outline procedures for the legal system. 


° The system allows for representation of the accused. 

® Conscientious objection is decriminalized. 

° The legal system controls the release of personal information. 

° A professional corps of legal officers is established within the 


military (1.e..: A JAG corps). 


13 This is done since many countries possessing the various attributes that characterize sub-objective #2 
often achieve sub-objective #] simultaneously. One could argue that greater civilian control over the 
military, coupled with a U.S. style military justice system, ts an effective way to eliminate the potential for 
war crimes. 


85 


« Military persons are afforded the same protections that civilians 
have in the country’s civilian judicial system. 


‘ Appeals are allowed. 


’ Court officers are both military officers and officials of the court. 


The prosecutor, defense council, investigator, and judge have legally 
established roles. 


Note: This attribute can be further divided into the following three 
attributes: 


a. Effective Advocacy. 
Indicators of Performance: 


. The accused 1s allowed to consult with a defense lawyer 
prior to the trial. 


4 Military and civilian legal sectors work together to ensure 
the rights of the accused are not violated. 


. Trial advocacy clinics are established for military lawyers. 


b. Fair, Impartial, and Authoritative Judgements. 


Indicators of Performance: 


° Judgements of the court are carried out. 

° Human rights executive training seminars are established 
for military judges. 

° Members of the military are openly willing to have their 


cases heard before a military court. 


oF Official criminal investigative procedures recognize the rights of 
the accused and accepted standards for the search and seizure of 
evidence. 
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Indicators of Performance: 


The investigative process 1s revised recognizing the rights of the 
individual. 


Interrogation and the potential for self-incrimination are 
constrained by law. 


The search and seizure of evidence 1s strictly regulated. 


The sentencing process 1s fair and impartial. 


Indicators of Performance. 


Detention practices are revised recognizing the right of due process 
and the assumption of innocence. 


Alternatives for punishment exist aside from prison. These include 
administrative punishment, nonjudicial punishment. and reduction 
in pay and grade. 


Humane military prison system. 


Indicators of Performance: 


No torture 1s allowed. 
Rules exist that enforce the humane treatment of prisoners. 


Outside agencies, such the International Red Cross, are allowed to 
inspect prison facilities. 


Corruption and the abuse of power are prevented and ethical behavior is 
promoted. 


Indicators of Performance: 


No undue command influence is exercised by the military. 


Tampering with the judicial process is made illegal. 
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e Rules of engagement (ROE) are developed during military 
operations that respect the sanctity of human life and the role of 
non-combatants. 


Additional Sub-Objective #1: 
Civilian Control Over the Military 
Specific Attributes: 


1. Elected officials control the military judicial system. 


Indicators of Performance: 


. The military justice system originated with the civilian authority 
that controls the military. 


e Members of the military have the right to bring unresolved 
problems to the attention of their elected representatives. 


Note: The remaining attributes for this additional sub-objective are the 
same as those described in the second section of this chapter. 


The complete hierarchy of sub-objectives and specific attributes for the 
improved military justice systems objective can be found in Figure 9. 

b. Limitations 

There are limitations to the use of the researcher's performance indices. 
The chief limitation is the selection of the U.S. military justice system as the model of 
choice. While few argue that the U.S. system of justice is widely regarded for its respect 
for human rights, the assumption that all countries can or must adhere to the same judicial 
procedures may be problematic. Customs and traditions may limit the strict application 
of the U.S. judicial model in some countries. Application of all the performance 


indicators identified in this study may not be possible in some countries. 
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Additionally, the proposed performance indicators for improved military 
justice systems only measure structural and process changes, not behavioral ones. 
Measuring changes in behavior involves highly subjective processes and is not possible 
with the researcher’s proposed indicators. A change to the military’s justice system is, 
however, the first step toward changing the judicial culture of the military. 

3. Defense Resources Management 

The third E-IMET objective is the improvement of defense resources 
management. The need for better management of military resources was recognized long 
before Congress expanded the IMET program to include respect for civilian control over 
the military and human rights. The Defense Resources Management Institute (DRMI), 
for example, was established at the Naval Postgraduate School in 1965 to help the United 
States and its allies with resources management education (DRMI Website, 1998). It is 
important to note that programs such as those offered by DRMI do not teach job-specific 
skills. Instead, they focus on building management skills that participants can apply 
throughout their careers. In order to propose performance indicators for this objective, 
however, one must clearly define the specific attributes of defense resources manage- 
ment. 

a. Defining Defense Resources Management 

While defining civilian control and humane military justice is fairly 
straightforward, specifying the attributes of improved resource management is much 
more difficult. To date, neither the State Department nor DSCA has clearly defined the 
E-IMET objective of “improved defense resources management.” While the management 
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literature does attempt to define the domain of resources management (see Hitch and 
McKean 1967, Sandler and Hartley 1995) DSCA and the Department of State have not 
specified the final result desired (also known as “endstate”) for their training and 
education efforts (Bonsper Interview, 1998).!4 In other words. while the endstates of the 
first two E-IMET objectives 1s clear (namely civilian dominance over the military and the 
adoption of a Western-style military justice system) the final result desired for the 
improved defense resources management objective is ambiguous. Without a clearly 
defined endstate one cannot identify the specific attributes of defense resources 
management. 

In order to develop the objectives hierarchy and performance indices for 
the third E-IMET objective the researcher proposes the following endstate for improved 


defense resources management training and education: 


The adoption of a decision-making process for allocating scarce 
defense resources that is transparent, interactive, and objectives- 
oriented. 


The researcher defines a “transparent” decision-making oe as one where critical 
decision-making steps are visibly linked, iterative, and follow a specific schedule. have a 
defined product, are completed by a specified stakeholder or group of stakeholders (and 
the completing entity 1s known to all the remaining stakeholders), and employ a well- 
defined and widely understood methodology. The opposite of a transparent process is a 
I4 The final result desired (or “endstate”) describes the desired result of the action (United States Marine 
Corps Warfighting, 1989). A clear explanation and understanding of the desired final result of defense 


resources management training and education is absolutely essential to the development of specific 
attributes and subsequent performance indices. 
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“black box” where defense programs are the outcome of some unknown military planning 
process (Bonsper, 1998). A decision-making process is “interactive” if both military and 
civilian stakeholders play an active role during each step in the process. Finally, it-is 
“objectives-oriented”’ if it evaluates existing programs and develops new ones based on 
continuously revised strategic objectives. 

Using the researcher’s proposed endstate, the sub-objectives and specific 
attributes for the E-IMET objective can now be identified and arranged into an objectives 
hierarchy. 


b. Measuring the Performance of Defense Resource Management 
Programs 


The E-IMET objective of improved defense resource management can be 


broken down into the following three sub-objectives: 


Sub-Objective #1: 
Transparent Resource Management Process 
Sub-Objective #2: 
Interactive Resource Management Process 
Sub-Objective #3: 


Objectives-oriented Resource Management Process 


The following specific attributes, along with their respective indicators of 
performance, have been identified for the sub-objectives of improved defense resources 


management: 
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Sub-Objective #1: 
Transparent Resource Management Process 
Specific Attributes: 


Decision-making steps are visibly linked, iterative, and follow a specific 
schedule. 


Indicators of Performance 


° A comprehensive threat assessment 1s performed and precedes 
defense resource allocation decisions. 


* A national military strategy is developed to counter the anticipated 
threat and precedes the development of specific programs. 


° Defense programs are developed to carry out the national military 
Strategy. 

* Defense program requirements are translated into annual budget 
requests. 

e Threat assessments, strategic planning, and program development 


occur periodically according to a published master schedule. 
Each decision-making step results in a defined product. 


Indicators of Performance 


. Threat assessments result in official strategic planning guidance 
issued to the military. 


. A national military strategy is published that defines strategic 
objectives and priorities. 


e Defense programs consist of detailed manpower (civilian and 
military) and material (equipment) requirements. 


. An annual budget ts produced. 


FS) 


Decision-making steps are completed by a specified stakeholder or group 
of stakeholders. 


Indicators of Performance 


e Civilian and military intelligence communities cooperate to 
produce threat assessments. 


* Individual service chiefs provide input to the national military 
Strategy. 
. Defense programs are developed independently by each service, 


but are reviewed by a central authority. 


" Military comptrollers translate program requirements into annual 
budget requests. 
& The people’s elected representatives approve disbursements from 


the national treasury for defense programs. 


« Civilian authorities approve the transfer of resources among 
programs. 


Each step in the decision-making process employs a well-defined and 
widely understood methodology. 


Indicators of Performance 


e Threat assessments update past threats, reassess current threats, and 
explore future threats. 


e The national military strategy defines the military’s plan for 
countering current and future threats and 1s not fiscally constrained. 


. Defense program choices are compared to each other on the basis 
of cost and effectiveness (capability) and ranked in order of 


importance (priority). 


« The planning process is capable of determining the relative 
importance of defense programs. 


e Criteria exist to evaluate, rank, and prioritize programs. 
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Civilian and military leaders use cost estimations and capability 
assessments to compare individual programs before selecting a 
preferred alternative. 
Budget requests are compared to resource availability. 
Priority programs are funded first when military program requests 
exceed resource availability (and funding cuts start with the lowest 
priority programs). 

Sub-Objective #2: 


Interactive Resource Management Process 


Specific Attribute: 


Military and civilian leaders share in the responsibilities of the resource 
management process. 


Indicators of Performance 


Military officials assess the threat, develop the national military 
Strategy, estimate the requirements of that strategy, and develop 
programs to package and execute the strategy. 


The people’s elected representatives approve the programs along 
with their respective cost ceilings. 


Inter-agency committees composed of military and civilian 
officials exist to coordinate planning and budgeting (1.e.: a 
“corporate” planning board overseas the entire resource 
management process). 


Sub-Objective #3: 


Objectives-oriented Resource Management Process 


Specific Attributes: 


Individual defense programs can be linked to specific strategic objectives. 
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Indicators of Performance 


The military develops a strategic plan. 


Defense programs are linked to objectives within the military’s 
strategic plan. 


The military justifies its major programs to the people’s elected 
representatives on the basis of strategic objectives. 


The resource management process balances short-term objectives with 
long-term objectives. 


Indicators of Performance 


Civilian officials allocate resources to maintain ongoing operations 
as well as to finance investments in procurement, new weapons 
research, and military construction. 


Long-term investment programs (procurement, construction, 
RDT&E) are fully funded while ongoing operations are 
incrementally funded. !> 


The development of future programs and the formulation of the 
next budget occur simultaneously (“parallel processes”) within the 
resource management process. 


Civilian and military officials consider long-term trends in threat 
conditions before making immediate program changes. 


The level of military program funding rises and falls with the threat 
environment. 


Defense programs are reevaluated when strategic objectives are revised. 


15 “Fully funded” means that the civilian appropriators have provided all the resources necessary to 
complete the approved long-term project and are committed to seeing the project through to its completion. 
It does not mean that the military receives all the resources at once, however. “Incremental funding”’ is the 
annual appropriation of only one year’s worth of resources. 
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Indicator of Performance 


e Periodic cost-effectiveness evaluations of existing defense programs based on 
revised strategic objectives. 


The complete hierarchy of sub-objectives and specific attributes for the 
improved defense resources management objective can be found in Figure 10. 

c Limitations 

Like the first two E-IMET objectives, there are limitations to the use of the 
researcher's performance indices. One possible limitation 1s the use of a generic 
Planning, Programming, and Budgeting System (PPBS) model as the template for this 
analysis. The PPBS template provides a past definition of defense resource management. 
Even the United States, considered a leader in defense resource management initiatives. 
has used more recent processes to manage its defense resources, including Management 
by Objectives (1973) and Zero-Base Budgeting (1977) (GAO. 1997). Although the PPB 
system has its detractors and flaws (see Roberts 1993), it is useful to demonstrate how 
objectives can be derived even 1f one were to substitute another process for PPBS in the 
future. Therefore, even though the use of a generic PPBS model may be a limitation. the 
researcher believes it is an excellent starting point for evaluating defense resources 
management. 

A more important limitation involves the actual measurement of 
progressions or improvements away from the defense resources management baseline 


assessment. In order to observe the indicators of performance security assistance officials 
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10. 


Figure 


would have to gain access to the highest levels of all ministries where defense resource 
allocation decisions are made. In the absence of such direct high-level access. actual 
improvements in defense resources management could be inferred from informal 
discussions with government officials at all levels. public statements regarding defense 


matters, and general coverage in the media of the country. 


D. THE BALANCED SCORECARD 

Assuming a country receives training in all three E-IMET areas, collecting data on 
every indicator will likely be time consuming. expensive. and overwhelming for the 
security assistance manager. I[n-country security assistance officials must choose a 
handful of indicators to apply to their unique situation. In order to present the various 
indicators of performance for each E-[MET objective in a user-friendly manner the 


researcher proposes the “balanced scorecard” format developed by Kaplan and Norton. 


1. The Kaplan/Norton Balanced Scorecard 


The Balanced Scorecard is simply “a set of measures that gives top managers a 
fast but comprehensive view of the business” (Kaplan and Norton. 1992, p. 1). Kaplan 
and Norton developed the Balanced Scorecard after realizing that senior private sector 
executives do not rely on only one set of performance measures to the exclusion of 


another (Kaplan and Norton, 1992). 


No single measure can provide a clear performance target or focus 
attention on the critical areas of the business. Managers want a balanced 
presentation of both financial and operational measures (Kaplan and 
Norton, 1992, p. 71). 
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Think of the Balanced Scorecard as the dials and indicators in an airplane 
cockpit. For the complex task of navigating and flying an airplane, pilots 
need detailed information about many aspects of the flight. [They need] 
indicators that summarize the current and predicted environment. (Kaplan 
and Norton, 1992, p. 72). 


Private sector managers realize that they cannot absorb the information generated 
by every conceivable measure of performance. “The Balanced Scorecard brings together, 
in a single management report, many of the seemingly disparate elements of a company’s 
competitive agenda” (Kaplan and Norton, 1992, p. 73). The Scorecard forces managers 
to choose among various performance measures with different business perspectives 
(objectives), focusing on the handful of measures that are most critical (Kaplan and 
Norton, 1992). By selecting three to four key measures within each business objective 
managers can link a wide range of performance measurement data allowing them to make 
better decisions in a time critical environment. The format of the Kaplan/Norton 
Balanced Scorecard, as well as their private sector example of an electronics company, 
can be found in Figure 11. 


2. The E-IMET Balanced Scorecard 


While Kaplan and Norton use the Balanced Scorecard to present financial and 
operational data for private sector firms, the same method of presentation is useful for 
managers of public sector programs with multiple objectives. The Kaplan/Norton 
Balanced Scorecard format can be modified so that chosen performance indicators are 
listed alongside their respective sub-objective each of the E-IMET objectives. These 


modified scorecards can serve as comprehensive security assistance management tools 
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Format: 


i 


Electronics Company Example: 


FINANCIAL Perspective CUSTOMER Perspective 
(nn 


Responsive On-Time Delivery 
Cash Flow P V 5 
Supply 



























Survive 


Succeed 









Quarterly New Products Percent 
Sales Growth From New 


Products 


- 
OL 














Operating 
CS : a 

Income Preferred Share of Key 

Supplier Accounts’ 

Purchases 













Prosper | Increased ROE 


INTERNAL BUSINESS Perspective 


Measures 


Manufacturing Cycle Time 
Excellence 
Unit Cost 


Yield 


Design 
ProcuceLvyiLcy 





Figure 11. Kaplan/Norton Balanced Scorecard Format and Example 
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for tracking the achievement of each E-IMET objective within a given country. The 
Kaplan/Norton private sector Balanced Scorecard thus becomes the “(Country’s Name) 
Balanced Scorecard” for the E-IMET program. 

The researcher recommends that the selection of performance indicators for a 
specific country be done by State or Defense Department professionals with significant 
in-country experience (1.e.: A Security Assistance Officer (SAO) or other embassy 
official). Deciding what indicators of performance are most meaningful for their country 
is a subjective process. For example, to aid in the selection of the handful of civilian 
control indicators used within a specific country Paul Stockton of the Naval Postgraduate 
School’s Center for Civil-Military Relations recommends grouping the various attributes 
and performance indicators into two major categories; those that are essential to civilian 
control and those that are indicators of the quality or depth of civilian control.!© Since no 
in-country security assistance officials participated in this study, an accurate grouping 
according to Dr. Stockton’s recommendation is outside the scope of this study. Such a 
categorization would, however, help regional security assistance officers decide what 
performance indicators to include in their country’s balanced scorecard. 

A complete “(Country’s Name) Balanced Scorecard” example can be found in 


Figure 12. 


16 Dr. Stockton identifies a third category as well; contentious indicators of civilian control. This category 
includes controversial indicators that are often debated among civil-military relations scholars. The 
researcher discussed this category in the indicators of civilian control over the military limitations section. 
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CIVILIAN CONTROL OVER THE MILITARY IMPROVED MILITARY JUSTICE SYSTEMS 


SUB-OBJECTIVE | Indicators of ADDITIONAL 
Performance OBJECTIVE 





indicators of 
Performance 













Military Used | Implementation of Establish a Appeals are Allowed 

















eo Counter Military Doctrines Western 

External Stressing External Style 

Threats Roles and Missions Military The Accused is 
Justice Allowed to Consult 






With a Defense 
Lawyer 


Redeployment of 
Military Away From 
Internal Uses 


System 









The Investigative 
Process is Revised 
Recognizing the 
Rights of the 
Individual 


Revised curriculum 
at service 
academies 







IMPROVED DEFENSE RESOURCE 
MANAGEMENT 


SUB-OBJECTIVE Indicators of 
Performance 


Decision-making A comprehensive 
Steps are threat assessment 


Visibly linked, 1 1s performed 

Iterative, and 

Follow a specific] A national 

Schedule military strategy 
Is developed 


Defense programs 
Are developed 
Based on the 
National military 
Strategy 





Figure 12. “(Country’s Name) Balanced Scorecard” Example 


E. CHAPTER SUMMARY 
Objectively measuring the results of the E-IMET effort involves tracking the 


changes within a country toward greater civilian control over the military. more humane 
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military justice systems, and improved defense resources management. In order for 
officials to determine whether the E-IMET intervention is responsible for significant 
organizational and national changes a baseline assessment must first be compiled. Due.to 
the multi-faceted nature of resource management, the State Department and DSCA must 
also clearly define a desired endstate for “improved defense resources management.” 
Various indicators of performance can then be combined with survey data so security 
assistance officers can compare post-training changes with the baseline assessment to get 
a clearer picture of the effectiveness of E-IMET training. While limitations do exist, the 
performance indicators proposed in this chapter offer an alternative to traditional 


anecdotal evidence. 
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VII. CONCLUSION 


A. INTRODUCTION 
In 1991, the U.S. Congress expanded the scope of the International Military 


Education and Training (IMET) program to include a focus on (Cope, 1995): 


. Greater respect for and grasp of democracy and civilian rule of law. 
including the principle of civilian control over the military: 


e Military justice systems in a democracy, including a better understanding 
of internationally recognized human rights: 


. Responsible defense resource management. 


In doing so, they aligned the IMET program with the new Post Cold War security 
environment. The Departments of State and Defense, however, did not agree on a system 
to evaluate the effectiveness of the expanded IMET (E-IMET) program. In 1993, the 
U.S. Congress enacted the Government Performance and Results Act requiring all 
government agencies to establish top-level goals and objectives, define how they intend 
to achieve those goals, and demonstrate how they will measure agency and program 
performance tn achieving those goals (GPRA, 1993). 

The purpose for this study is to address how the Defense Security Cooperation 
Agency (DSCA) and the Department of Defense (DoD) can tackle the challenge of 
measuring the effectiveness and performance of the E-IMET program. This chapter will 


summarize the study's findings based on the original research questions. 


B. RESTATEMENT OF RESEARCH QUESTIONS 
Primary Research Ouestion: 


What could a performance evaluation system developed for the E-IMET program 
look like? 


Subsidiary Research Questions: 

l. What are the challenges to performance evaluation of public programs? 

2 What factors unique to the political environment necessarily limit the 
application of rational-analytic assessment models when measuring costs 
and benefits of current security assistance programs? 

Can a system be developed that balances the reality of the political 


bureaucratic process with the rationality of economic cost-benefit 
analysis? 


o>) 


4. What are the implications of implementing a performance evaluation 
system on current security assistance programs? 


C. SUMMARY OF FINDINGS 
| Challenges to Public Sector Performance Evaluation 
The obstacles to public sector performance evaluation are primarily institutional, 
technical, and financial. Each of these obstacles presents a significant challenge to E- 
IMET performance evaluation. 
a. Institutional Obstacles 
Institutional obstacles include the influence of political pressure on 
performance measurement, the effect of multiple self-interested constituencies on 


objectives consensus, and the lack of experience of government managers in dealing with 
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performance data. Each of these obstacles limits the ability of E-IMET officials to 
measure the effectiveness of the E-IMET program. 

The E-IMET program has four diverse groups of stakeholders. all sharing 
in the management of the E-IMET program. The presence of so many powerful 
stakeholder groups makes the setting of a single set of security assistance objectives 
difficult. The Executive Office of the President sets national security assistance direction 
while the Congress, driven by their commitment to the Government Performance and 
Results Act, appropriates the funds necessary for the programs. Each has a different 
political stake in the performance of the E-IMET program. The Departments of State and 
Defense share responsibility for the management of the E-IMET program. but have 
potentially different financial stakes in the measurement of the program's effectiveness. 
Getting these groups to agree on a single set of objectives for security assistance and the 
E-IMET program is a challenge. 

Compounding this problem is the fact that State and Defense Department 
official documents do not separate objectives from strategies. Fortunately. once the 
researcher separated objectives from strategies a common set of national security 
assistance objectives emerged. This consensus forms the objective baseline for future 
performance evaluation. 

b. Technical Obstacles 

Technical obstacles include differentiating between output and outcome 


measures, the need for several measures of performance for any one program. the lack of 
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standards for judging governmental performance, and the lack of timely performance 
feedback. 

Due to the multi-faceted nature of the E-IMET program no single measure 
of performance is adequate. E-IMET officials must develop and implement multiple 
measures of performance simultaneously in order to provide Congress with the data 
required by the Results Act. Even with such a system, no standards exist that indicate 
whether the E-IMET intervention is “effective enough’ to merit increased funding. 
Periodic performance reporting must be compared with an objective baseline assessment 
of a country’s political. judicial, and managerial characteristics (collected before the first 
E-IMET intervention) in order to attribute specific results to the E-[MET intervention. 

Additionally, security assistance programs like E-IMET are often long- 
term intervention efforts. Such efforts often do not produce immediate results and are 
therefore not conducive to the immediacy of the GPRA reporting requirement. 

rig Financial Obstacles 

Implementation of performance measurement systems can be very costly. 
stretching the availability of financial and human resources. Systematically tracking the 
progression toward civilian control over the military, Western-style military justice 
systems, and improved defense resource management within a country requires the 
commitment of State and Defense Department assets. Funding for the IMET and E- 
IMET programs 1s already tiny relative to other discretionary programs. Implementing a 
comprehensive E-IMET performance evaluation system would likely absorb the very 
funding increases that the system is designed to capture. A comprehensive performance 
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measurement system implemented under current budgetary constraints would be difficult. 
DoD and DSCA must therefore adopt a performance evaluation system that is easy to use 
and can be implemented without additional funding. 

Zz Factors Limiting Rational-Analytic Assessment 

The chief factor limiting the objective assessment of E-IMET effectiveness is the 
inability to control] external variables. It 1s difficult to separate the effects of the external 
environment from the E-IMET intervention. Security assistance officials will likely 
never be able to establish a concrete cause and effect relationship between the E-IMET 
intervention and a given result due to the presence of uncontrollable political and 
economic forces. They can, however, do a better job demonstrating to Congress a 
country’s post-intervention progression toward the objectives of the E-IMET program. 

Technically measuring the effects of the E-I[MET program is also difficult. 
Quantifying the benefits that result from greater civilian control over the military, 
improved military justice systems, and more responsible defense resource management Is 
very subjective. Further complicating this process 1s the lack of a clear definition of 
improved defense resources management. Neither DSCA nor the Department of State 
has clearly defined a desired endstate for the improved defense resource management 
objective. While the attributes of civilian control over the military and Western-stvle 
military justice systems are much easier to describe, the attributes of improved defense 
resource management are more ambiguous. 

Another important limitation to the performance measurement process is the 
disjointedness of the government's security assistance organizations. In addition to the 
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obstacles created by the presence of multiple constituencies, the current bureaucratic 
structure of the security assistance community is a major impediment to the measuring 
process. The researcher agrees with a 1995 National Defense University study that 


observed: 


The apportionment of responsibilities—system oversight (State), program 
management (Defense), and program requirements (Unified Commands, 
program teams, and foreign governments)...1s inherently dysfunctional, 
creating the impression that IMET is a program without long-range 
benefit, one well-suited only for near-term displays of U.S. attention and 
associated “carrot and stick” diplomacy. This leads many in government 
to conclude that IMET is an expendable program (Cope, 1995, pp. 52-53). 


The report added that: 


Although international military education and training is a diplomatic- 
defense, or political-military, instrument of national security policy, 
INSS's research found substantial agreement that the DoD has the greatest 
interest in and commitment to the IMET program. For the last 19 years, 
the State Department has dominated the process, while remaining 
ambivalent toward it, whereas Defense, without active sustained interest 
by staff principles, generally has deferred to State (Cope, 1995, p. 53). 


The results of this study show that little has changed within the State and Defense 
Departments since 1995. Sharing responsibility for the IMET program and its expanded 
initiative between two major bureaucratic government agencies guarantees that perform- 
ance measurement will fail to achieve its potential. Until the Departments of State and 
Defense streamline the program’s chain of responsibility and authority E-IMET 


performance evaluation will continue to be limited. 
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3. The E-IMET Performance Evaluation System 

A spectrum of approaches to the evaluation of public programs exists within the 
performance evaluation literature. Anchoring one end of the spectrum is the “technically 
rational paradigm’ that treats performance measurement as an objective. scientific aid to 
decision-making. Anchoring the opposite end of the spectrum is the “politically rational 
paradigm” that emphasizes the effect of political power on the measurement of 
performance. Given the difficulties of technically measuring the effects of the E-IMET 
program, the politically rational paradigm was used to develop the following four-step E- 


IMET performance measurement process: 


I. Identify the Key E-IMET Stakeholders; 


2. Identify the Objectives of U.S. Security Assistance and the E-IMET 
Program: 

= Develop Performance Indices for the E-IMET Objectives; 

4. Apply technically rational performance measurement techniques. 


Only steps #1-3 were completed during this study. Due the researcher's lack of 
access to country-specific E-IMET training data and security assistance officials. step #4 
was beyond the scope of this study. In order to apply the E-IMET performance indices 
proposed in Chapter VI], DCSA must first conduct an objective baseline assessment in 
each country that receives E-IMET training. DSCA must then allow the in-country 
Security Assistance Officers (SAOs) to select the indicators of performance that are most 


applicable to their country’s current situation. Systematic reporting of progressions away 
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from the baseline assessment must follow in order to establish a stronger cause and effect 
relationship and meet the requirements of the Results Act. 
4. Implications of the E-IMET Performance Evaluation System 
Successfully measuring the performance of the E-IMET program has both internal 
and external implications for organizations like DSCA. 

a. Internal Implications 

Requiring SAQs to systematically report on the progress toward greater 
civilian control over the military, improved military justice systems, and improved 
defense resources management is critical to the performance measurement effort. 
Information is only as good as its source, however. SAOs must be trained to recognize 
the indicators of performance and disciplined enough to consistently report them. 
Additionally, DSCA must be organized to receive the data and possess the processing 
capabilities necessary for the collection and dissemination of the information. 

To date, security assistance officials have only been required to present 
anecdotal evidence to guarantee a future for their programs. Informal student opinions, 
the testimony of a key Unified Commander, and the remarks of an Ambassador or foreign 
official are examples of the anecdotal evidence used by security assistance officials to 
justify their programs. But the Results Act requires organizations like DSCA to 
demonstrate the results achieved. The application of a mgorous performance 
measurement system may be outside the experience of most security assistance managers. 
Understanding the limitations and challenges of a public sector performance evaluation 


system is crucial to making one work. Performance measurement systems like the one 


designed in this study must be fully integrated with the strategic management of an 
organization. Inexperienced security assistance officials might see the results of their 
performance evaluation efforts simply as a report card and not as the strategic 
management tools that they are. Successfully evaluating the performance of an E-IMET 
intervention will determine if it is having the intended effect but it will not tell DSCA 
officials how to change it to make it more effective. 

b. External Implications 

The external implications of measuring the performance of security 
assistance programs are mixed. If the results are not favorable should one conclude that 
efforts to export democratic ideals are failing and should be stopped in order to save only 
3% of the federal budget? Exporting democracy through security assistance training has 
traditionally been an effort based on faith—*train them and they will change.” 
Demonstrating results is foreign to the nature of security assistance program managers 
who do not discount non-quantifiable factors. Meeting the requirements of the Results 
Act could help solidify future congressional support but the emphasis on visible results 
might overshadow invisible ones. An imperfect E-IMET performance evaluation system 
must be used to reinforce the testimony of international affairs experts. not as a 


replacement for informed opinion. 


D. RECOMMENDATIONS 


h. Recommendations for Further Action 


It is recommended that DSCA consider implementing the use of the performance 


indicators proposed in this study to measure the effectiveness of their E-IMET 


intervention efforts. To do so, DSCA and in-country security assistance officials must 
build a country-specific Balanced Scorecard of performance indicators for each of the E- 
IMET objectives. A standard baseline assessment format must also be developed to 
provide the comparative foundation from which to measure the effectiveness of the E- 
IMET intervention. The training of the SAOs to recognize the indicators of performance 
and properly complete the baseline assessment must be a prerequisite of the system’s 
implementation. Information resulting from the systematic reporting of progressions 
away from the baseline must then be gathered over time and presented as evidence for 
future funding decisions. 

The researcher also recommends that DSCA and the Department of State clearly 
define an endstate for “improved defense resources management.” A final result desired, 
as well as the desired attributes, must be communicated throughout DSCA. DSCA 
should solicit the help of Naval Postgraduate School academic professionals to build 
upon the researcher’s definition and objectives hierarchy for improved defense resources 
management. 

Zz Recommendations for Further Study 

This study was completed without the input of regional experts. Building a 
Balanced Scorecard of E-[MET performance evaluation indicators requires a broad 
understanding of a country’s civil-military traditions, existing military justice system, and 
defense resource management practices. The indicators of performance proposed in this 


study must be refined to fit the unique situations of specific countries. A baseline 


114 


assessment format must also be developed so that future progressions toward the 
objectives of the E-IMET program can be measured. 

This study also did not consider alternatives for the delivery of the security 
assistance service. The researcher developed performance indices for the program as it 
currently exists. Now that performance indices have been proposed a cost-effectiveness 
comparison of alternatives is possible. Other alternatives for delivering security 
assistance training should be explored to determine the most efficient and effective 
method. 

While not specifically addressed during this study, there are implications that the 
current apportionment of security assistance management responsibilities within the 
national government is inherently dysfunctional and basic IMET relationships need 
adjustment. DoD and DoS may need to streamline IMET management and budget 
planning, create a dedicated interdepartmental review council to oversee the entire 
program, and review the overarching policies that guide foreign military education. 
Further study 1s needed to determine whether current organizational relationships are 
appropriate and to determine the optimal apportionment of IMET responsibilities and 


authority. 
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APPENDIX. DEFINITIONS 


The following terms have been defined as listed below for the purposes of this study: 
Baseline Data: Initial collection of data to establish a basis for comparison. 


Benchmark: A standard or point of reference used in measuring and/or judging quality 
or value. 


Effectiveness: The production of a decided, decisive, or desired effect. 
Efficiency: The ratio of outputs to inputs in a process. 


Emerging Democracy: A formerly non-democratic national government that seeks to 
implement broad-based democratic reforms. Examples include Poland, Hungary, and 
various former Soviet Republics such as Estonia, Latvia. and Lithuania. 


External Stakeholder: An individual or group outside the boundaries of the producing 
organization or program that receives or uses the output of the process. 


Internal Stakeholder: An individual or group inside the boundaries of the producing 
organization or program that receives or uses the output from a previous stage or process 
to contribute to production of the final product or service. 


Measure: One of several measurable values that contribute to the understanding and 
quantification of a performance indicator. 


Measurement System: A system of practices, procedures, criteria, and standards that 
govern the collection of data (input), the analysis of the data (throughput). and the 
compilation of the results into quantitative or qualitative forms (output). 


Mobile Education Teams (METs): A group of American-led professionals tasked to 
travel to a host country to conduct E-IMET courses. It should be noted that “METs— 
differ from the more commonly known military “MTTs” (Mobile 7raining Teams). 
METs are primarily civilian educators that teach military-to-civilian training courses. 
where MTTs are primarily military personnel who deliver military-to-military training. 


Objectives Hierarchy: A useful approach to systematically link key stakeholders: and 
their respective objectives to performance indicators. A hierarchy works down from a 
higher level objective to relevant sub-objectives, to specific attributes that characterize 
those sub-objectives, and finally to measurable characteristics that allow for the 
quantification of the desired attributes. 
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Outcome Measurement: An assessment of the results of a program activity as 
compared to its intended purpose. Outcome refers to the impact on society of a particular 
public sector activity and is closely related to the concept of effectiveness. 


Output Measurement: The tabulation, calculation, or recording of an activity or effort. 
Output measurement traditionally refers to the quantity of the good or service provided, 
and is closely related to the concept of efficiency. 


Paradigm: A systematic statement of the basic assumptions, concepts, and propositions 
employed by a school of analysis. 


Performance Goal: A target level of an activity expressed as a tangible measurable 
objective, against which actual achievement can be compared. 


Performance Indicator: A particular value or characteristic used to measure output or 
outcome. 


Performance Management: The use of performance measurement information to help 
set agreed-upon performance goals, allocate and prioritize resources, inform managers to 
either confirm or change current policy or program directions to meet those goals, and 
report on the success in meeting those goals. 


Performance Measure: A quantitative or qualitative characterization of performance. 


Performance Measurement: A process of assessing progress toward achieving 
predetermined goals, including information on the efficiency with which resources are 
transformed into goods and services (outputs), the quality of those outputs (how well they 
are delivered to clients and the extent to which clients are satisfied), and outcomes (the 
results of a program activity compared to its intended purpose), and the effectiveness of 
government operations in terms of their specific contributions to program objectives. 


Performance-based Budgeting: Budgeting initiatives designed to measure performance 
and link resources with results. The latest iteration in a series of Federal Government 
performance budgeting reforms 1s the Government Performance and Results Act (GPRA) 
of 1993. 


Political Rational Paradigm: A paradigm that emphasizes the effect of political power 
on the measurement of performance. 


Process-Oriented Measurement System: A measurement system whose purpose is to 
assess the attainment of the desired results, or outcomes, based on defined agency 


objectives. 


Program Performance Measurement: The regular collection and reporting of a range 
of data including inputs, workload levels, outputs, and outcomes. 
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Results-Oriented Measurement System: A measurement system whose purpose is to 
assess the efforts to reengineer agency processes in order to achieve greater efficiency. 


Security Assistance: The transfer of military and economic assistance to foreign 
governments through sale, grant, lease. or loan. 


Stake: A claim on the organization or program that each stakeholder has. A stake may 
be tangible or intangible, explicit or implicit. A stake may also be political. economic. 
social, or psychological in nature. 


Stakeholder: Any person, group, or organization that can place a claim on, or influence. 
the organization’s or program’s resources, outputs. or outcome: is affected by those 
outputs; or has an interest or expectation of the organization or program. 


Strategic Goal: A long-range change target that guides an organization's efforts in 
moving toward a desired future state (also referred to as an “end” ). 


Strategic Objective: A broad time-phased measurable accomplishment required to 
realize the successful completion of a strategic goal. 


Technical Rational Paradigm: A paradigm that treats performance measurement as an 
objective, scientific aid to decision-making. Central to the application of this paradigm is 
the existence of objectives consensus and measurable costs and benefits. Examples 
include economic cost-benefit analysis. performance ratios. and linear programming 
models. 
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